Extensible heterogeneous multi-uav cooperative fire extinguishing system

Abstract

The present application relates to a scalable heterogeneous multi-unmanned aerial vehicle cooperative fire extinguishing system. The system adopts a scalable open architecture, describes the required interface at the task level through a platform-independent IDL interface description file, and parses the interface description file using the CORBA specification, so that the unmanned aerial vehicle can cooperate to complete the forest fire monitoring, positioning and accurate fire extinguishing. In the future, the cooperative fire extinguishing system can be applied to various multi-unmanned aerial vehicle cooperative scenarios that cannot be completed by a single unmanned system. For the task scenario of multi-unmanned aerial vehicle joint forest fire automatic monitoring, positioning and fire extinguishing, the present application designs the required scalable, inherited and modular general interface description in the task level through the IDL language in the task scenario, parses it using the CORBA specification, generates the corresponding stub and framework files of various unmanned system platforms, and realizes the platform-independent, fast technology insertion and scalable framework for forest fire monitoring, positioning and fire extinguishing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fire extinguishing systems, and particularly relates to an extensible heterogeneous multi-unmanned aerial vehicle cooperative fire extinguishing system. BACKGROUND

[0002] To ensure that the autonomous unmanned aerial vehicle adapts to environmental or configuration changes, the flight control system of the autonomous unmanned aerial vehicle must have highly complex performance, such as the ability to reconfigure the control algorithm, plug-and-play extensibility for new technologies, interoperability between different components (different components running on different processors), and an open software structure that can be extended by users to provide new or extended functions. These performances will ensure the openness and adaptability of the system, that is, the system can flexibly integrate any other tool developed resources and can be dynamically reconfigured to quickly adapt to changes in situations. These requirements are a great challenge to the designers of the flight control system of the autonomous unmanned aerial vehicle, and many flight control systems developed at present have great limitations. The closed / special nature of the system makes data exchange complex and inflexible, limiting hardware changes and software reconfiguration. In this regard, many institutions at home and abroad have carried out a large number of simulations and tests, and the research on the single technologies and algorithms (such as visual perception, trajectory planning, navigation, and intelligent control) required in the autonomous unmanned flight control has been relatively mature.

[0003] Although the current autonomous unmanned flight control system has certain service functions, there are still the following main problems: there are many system components, and there are various differences between the components, including different hardware devices, operating systems, programming languages, communication protocols, etc. A lot of work is needed to coordinate the work of these components; the devices are tightly coupled, and adding or replacing a device means adjusting many existing devices, which includes both hardware adjustment and software adjustment, functional expansion, code modification, etc. to the original software, and also involves a lot of repeated testing work of the existing software, which reduces the development efficiency and system stability; the connection relationship between the devices is complex, there are Ethernet connection, 1553 bus connection, and serial port connection, different connection relationships have different access methods and addressing methods, and the complex connection relationship leads to complex data control flow, increases the difficulty of software development and testing workload, and prolongs the development cycle. At present, the biggest difficulty is the lack of a general software platform to realize the integration and real-time distributed communication of each subsystem.

[0004] CORBA(Common Object Request Broker Architecture) is a standard object-oriented application system specification formulated by OMG(Object Management Group), which provides a unified interface to support the formation of such a general software platform, and provides a software bus. The CORBA architecture is a solution proposed by OMG to solve the interconnection problem of hardware and software systems in the distributed processing environment (DCE). Hardware modules can be regarded as objects, and the use of CORBA can solve the interface unification problem.

[0005] For example, the prior art patent document with publication number CN102081545A proposes a method for implementing a software communication architecture (SCA) on an embedded platform, using CORBA middleware as a software bus, using XML files as information middleware, and constructing a framework software on a PowerPC and DSP embedded platform; using a component engine (CE) to assist the tools for developing SCA, software modeling and hardware modeling of the system; using a CORBA compiler to compile the IDL code of the core framework, generating a stub file and a framework file of the interface, and implementing a servo class of the framework interface; using CE software to generate an XML domain description file for the application model and the platform model of the system; improving and extending the code generator of CE, writing a code template for the device and the program framework, implementing a component program and a device program, running the application component program and the device program on different nodes, and testing the correctness of the waveform function.

[0006] Based on the technical requirements of the unmanned autonomous flight system, the purpose of the present application is to establish an unmanned autonomous flight system that realizes the integration of each subsystem and real-time distributed communication based on the CORBA architecture.

[0007] In addition, on the one hand, there are differences in understanding of the technical personnel in the art; on the other hand, the applicant has studied a large number of literatures and patents when making the present application, but due to the limitation of space, all the details and contents have not been listed in detail, but this does not mean that the present application does not have the characteristics of these prior arts, on the contrary, the present application has all the characteristics of the prior art, and the applicant reserves the right to add relevant prior art in the background art. SUMMARY

[0008] In view of the deficiencies of the prior art, the application provides an extensible heterogeneous multi-unmanned aircraft cooperative fire extinguishing system. The system generates an IDL interface description specification file set according to an interface description class, and parses the IDL interface description specification file set through a CORBA specification to generate a framework for a server and a stub for a client for different operating systems and programming languages in a plurality of components. A first component calls an interface provided by a second component and controls the second component through the interface. The components at least include a cruising unmanned aircraft, a fire extinguishing unmanned aircraft and a console. For example, the first component can be the cruising unmanned aircraft, and the second component can be the fire extinguishing unmanned aircraft.

[0009] The cooperative fire extinguishing system provided in the application adopts an extensible open architecture, describes the required interface at a task level through a platform-independent IDL interface description file, and parses the interface description file through a CORBA specification, so that the unmanned aircrafts can cooperate to complete wild fire monitoring, positioning and accurate fire extinguishing. In the future, the cooperative fire extinguishing system can be applied to a plurality of multi-unmanned aircraft cooperative scenarios that cannot be completed by a single unmanned system. For the task scenario of multi-unmanned aircraft joint forest fire automatic monitoring, positioning and fire extinguishing, the application designs an extensible, inherited and modularized general interface description required in the task scenario at a task level through an IDL language, parses the interface description through a CORBA specification, generates a stub and a framework file corresponding to various unmanned system platforms, and realizes a platform-independent, fast technology insertion and extensible framework for forest wild fire monitoring, positioning and fire extinguishing. The architecture allows future unmanned devices with stronger performance, higher computing power, better fire extinguishing efficiency and more accurate fire condition identification to be quickly added.

[0010] According to a preferred embodiment, the components include at least one unmanned aircraft, the unmanned aircraft is configured to realize flight positioning and navigation based on a UWB technology, and the unmanned aircraft calculates speed components (V x , V y ) thereof on a horizontal two-dimensional coordinate system according to the following formula:

[0011] V x =V*cos(tan -1 ((Y1-Y0) / (X1-X0))-(θ'+θ x ));

[0012] V y =-V*sin(tan -1 ((Y1-Y0) / (X1-X0))-(θ'+θ x ))。

[0013] Wherein, V is a cruising speed of the unmanned aircraft in a horizontal direction; θ' is an angle returned by an IMU sensor of the unmanned aircraft; θ xis the angle between the power-on unmanned aerial vehicle and the x-axis of the coordinate system; (X0, Y0) is the position of the unmanned aerial vehicle; and (X1, Y1) is the position of the target point.

[0014] According to a preferred embodiment, the unmanned aerial vehicle is configured to train a deep convolutional network using a training set of images taken in advance, and to use the trained deep convolutional network to identify a video stream to determine whether there is a fire or other unmanned aerial vehicle within a specified range.

[0015] According to a preferred embodiment, the unmanned aerial vehicle comprises at least a cruising unmanned aerial vehicle and a fire-extinguishing unmanned aerial vehicle, the cruising unmanned aerial vehicle is configured to preliminarily locate a fire by ignoring the distortion of the camera, and to instruct the fire-extinguishing unmanned aerial vehicle to move to the location of the fire at the initial stage of the fire, while the cruising unmanned aerial vehicle performs further accurate positioning.

[0016] According to a preferred embodiment, the cruising unmanned aerial vehicle is configured to calculate the preliminary positioning coordinates (X f , Y f ) of the fire according to the following formula, while ignoring the distortion of the camera:

[0017] X f =√((F y *(R y / C y )) 2 +(F x *(R x / C x )) 2 )*sin(tan -1 ((F y *(R y / C y )) / (F x *(R x / C x )))+θ'

[0018] +θ x )+X u ;

[0019] Y f =-√((F y *(R y / C y )) 2 +(F x *(R x / C x )) 2 )*cos(tan -1 ((F y *(R y / C y )) / (Fx *(R x / C x )))+

[0020] θ'+θ x )+Y u 。

[0021] wherein R x is the actual length of the picture taken by the gimbal camera on the cruising UAV at cruising altitude, R y is the actual width of the picture taken by the gimbal camera on the cruising UAV at cruising altitude; C x and C y are the number of pixels of the length and width of the camera respectively; (X u , Y u ) is the position of the UAV; (F x , F y ) is the pixel coordinate of the identified fire.

[0022] According to a preferred embodiment, the cruising UAV is configured to mark the center point of the fire in the picture according to a deep convolutional neural network, and to adjust the movement of the fire-extinguishing UAV according to the difference between the center point of the fire and the center point of the picture.

[0023] According to a preferred embodiment, the cruising UAV is configured to mark the center point of the fire in the picture according to a deep convolutional neural network, and to adjust the movement of the UAV according to the difference between the center point of the fire and the center point of the picture, and the difference in X-axis and the difference in Y-axis are both less than a preset pixel to determine that the fire is successfully located.

[0024] According to a preferred embodiment, the fire-extinguishing UAV is configured to throw the fire-extinguishing bomb to the location of the fire according to the identification result of the cruising UAV.

[0025] According to a preferred embodiment, the fire-extinguishing UAV is configured to calculate its speed components (V x2 , V y2 ) in the horizontal two-dimensional coordinate system according to the following formula:

[0026] V x2 = V x1 *cos((θ c +θ c ')-(θ a +θ a '))- V y1 *sin((θ c +θ c ')-(θ a +θ a '));

[0027] V y2 = V x1 *sin((θ c +θ c ')-(θ a +θ a '))+V y1 *cos((θ c +θ c ')-(θ a +θ a '))。

[0028] Wherein, θ c is the angle obtained by the IMU sensor of the cruise UAV; θ a is the angle obtained by the IMU sensor of the fire extinguishing UAV; θ a ' is the angle between the X axis and the fire extinguishing UAV when powered on; (V x1 , V y1 ) is the position of the fire extinguishing UAV in the coordinate system set by the camera of the cruise UAV; θ c ' is the angle between the X axis and the cruise UAV when powered on.

[0029] The application also provides an extensible heterogeneous multi-UAV cooperative fire extinguishing system, which comprises at least a UAV provided with a gimbal camera and a console provided at an operator end. The UAV comprises at least a data processing module, which is configured to: convert a first fire image with a first resolution obtained by the UAV at a first time into a second fire image with a second resolution lower than the first resolution and / or keep at least one third fire image with the first resolution, and transmit one or more of the second fire image, corresponding identification area information and at least one third fire image corresponding to the identification area information to the operator end at a second time. The identification area information can be recognized by the console at the operator end to perform image fusion and display of the information and / or the third fire image and the second fire image at the operator end. DETAILED DESCRIPTION

[0030] In order to facilitate the understanding of CORBA, the following describes some basic concepts involved in CORBA:

[0031] Object: An object system comprises a plurality of object entities. An object is an encapsulated entity capable of providing one or more services for client requests. In a distributed environment supported by an ORB (Object Request Broker), an application can be composed of many objects, and the functions of the application are realized through the interaction between the objects, and the interaction between the objects is transmitted through the ORB.

[0032] Request: A client requests a service by constructing a request, in CORBA parlance, the requester is called a client. The request specifies the object implementation that provides the service, the operation to be performed, the context of the operation, and the parameters to be used in the operation.

[0033] Object Reference: An identifier used to specify an object is called an object reference. An object can have multiple object references, but an object reference refers to only one object.

[0034] Object Implementation: In CORBA parlance, the provider of a service is called an object implementation. It uses an interface to describe the operations that can be performed on it.

[0035] Method: The executable code that corresponds to an operation described in an object implementation is called a method. Making a method execute is called method activation.

[0036] Activation: When a service request is executed, it causes a method in an object or object implementation to be invoked. If the initial state of the object or object implementation cannot satisfy the requirements of the method invocation, some means must be provided to change its state to satisfy the requirements of the method invocation. This process is called activation, and the reverse process is called deactivation.

[0037] Interface: An interface is a collection of related functions. Each function in an interface is fully specified, including the function name, the number of parameters, the type of parameters, the return type, and the exceptions that can be thrown. It is important to note that an interface only defines the prototype of a function, not the implementation. This provides flexibility in the implementation of the function. Interfaces are defined in a neutral interface description language. Although the IDL language only provides a conceptual framework for objects that are manipulated by ORBs, the ORB does not require the source code of the IDL at run time. As long as the equivalent information in the stub or run-time state of the interface library is available, the ORB can perform its functions in a specific manner.

[0038] Stub: A stub is actually a piece of program code that provides a dummy implementation for each operation (method) in an interface. A stub can be viewed as a mirror image of the real object in the client process, where the interface must be predefined, thus it provides a static way of calling for the client. From the client's point of view, the stub appears to provide a local implementation of the object - the client calls the operations on the object in the stub and gets the desired results at the end of the operation. The stub is not really responsible for the implementation of the object, it is just a "proxy" of the real object in the local side, which accepts the request from the local client, marshals it and passes it to the ORB core; when the remote operation returns, the proxy unmarshals the return parameters and results and returns them to the client in a way that the client can accept, making the client "believe" that the operation was implemented by the local "proxy" - the stub. The actual work of the stub includes accepting the request from the client program, returning the results of the operation, and marshaling and unmarshaling.

[0039] Skeleton: A skeleton is actually a piece of program code that provides a framework for writing the server implementation (i.e., object implementation) code for a given interface. In terms of function, a skeleton is like a stub on the server side, it is not responsible for the implementation of the object. The skeleton corresponding to a stub provides a static way of implementing for the server. The IDL compiler translates the IDL file describing the interface and generates the IDL stub and IDL skeleton for the specific programming language.

[0040] Inheritance: In object-oriented programming, a new class can be created by extending an existing class and inheriting the attributes and behaviors of the class.

[0041] Superclass: The class being inherited is generally called the "superclass" or "parent class", and the class doing the inheriting is called the "subclass". When a subclass inherits a superclass, it does not have to write out all of the instance variables and methods, it just has to declare that it inherits the instance variables and methods of the defined superclass. The superclass and subclass pictorially describe the hierarchical relationship of inheritance. Inheritance saves a lot of work in defining a new class, and it makes it easy to reuse code. For example, if a car is taken as the parent class, when a car subclass sedan is created, attributes such as brand, price, and maximum speed are automatically defined, and when the brake method is called, the brake method defined in the car class is automatically called. But a subclass does not have to use the inherited attributes and methods, a subclass can choose to override the existing attributes and methods, or add new attributes and methods.

[0042] CORBA is an application software architecture and object technology specification proposed by the OMG, whose core is a set of standard languages, interfaces and protocols to support the interoperability between heterogeneous distributed applications and the reuse of objects independent of platform and programming language. In order to invoke an operation of a distributed object, the client must know the interface provided by the object. An object's interface is composed of the operations it supports and the data types that can be passed to and from these operations. The client also needs to know the semantics and functionality of the operations.

[0043] In CORBA, object interfaces are implemented in the OMG Interface Definition Language (IDL). The sole purpose of IDL is to allow object interfaces to be defined in a form that is independent of any particular programming language. This consideration allows applications to be implemented in different programming languages to facilitate interoperability. In essence, IDL is a language used to describe the interface between a client object that generates a request for an object service and the object that provides the service. An IDL file describes the data types and method framework, and a servant object provides the data and methods for a specified object implementation.

[0044] An important feature of IDL interfaces is that they can inherit one or more other interfaces. This allows new interfaces to be defined from existing interface items, and objects that implement the new derived interfaces can substitute for those that support the existing base interfaces.

[0045] IDL provides one special case of inheritance: all interfaces implicitly inherit the Object interface defined in the CORBA module. This special base interface provides operations common to all CORBA objects.

[0046] An ORB is a middleware that establishes the Client / Server relationship between objects. Through the ORB, a Client can transparently reference a Server object on the same machine or on a network. The Server object can be local or on another machine connected through a network. The ORB intercepts the reference, locates the object that implements the service and passes the parameters to it, invokes the method and returns the final result. The Client is unaware of the location of the object, its programming language, its operating system and other system information that is not part of the object interface. Based on this, the ORB enables the interoperability between applications located on different machines in a distributed environment and the seamless connection between multi-object systems.

[0047] Forest fires are known for their large-scale destructive power and time sensitivity. When the best opportunity to extinguish the fire is missed, the forest fire will cause immeasurable damage to the environment and the safety of firefighters. For disaster-related decision-making, Boyd's OODA loop (Observe, Orient, Decide and Act, also known as Boyd cycle) plays an important role, and the speed of iterating the OODA loop is crucial for disaster search and rescue. In recent years, with the improvement of remote sensing ability, rapid decision-making ability and adaptation to harsh environments of air unmanned aerial vehicles (UAVs), air UAVs have gradually played an important role in automatic monitoring, positioning and extinguishing of fire. At present, the ability of a single UAV is becoming stronger, and an open and extensible general framework is needed to quickly realize the cooperation of multiple heterogeneous UAVs to complete complex tasks that cannot be completed by a single UAV.

[0048] To this end, for the task scenario of multi-UAV joint automatic monitoring, positioning and extinguishing of forest fires, the present application designs an extensible, inheritable and modular general interface description required in this task scenario at the task level through the IDL language, and uses the CORBA specification to parse it to generate corresponding stub and framework files for various unmanned system platforms, thereby realizing a framework that satisfies forest fire monitoring, positioning and extinguishing, is independent of the unmanned system platform, allows for rapid technology insertion and is extensible. This framework allows the rapid addition of future unmanned devices with stronger performance, higher computing power, better fire extinguishing efficiency and more accurate fire identification.

[0049] The system proposed in the present application adopts a heterogeneous multi-UAV cooperative fire extinguishing framework, which mainly includes an IDL interface description specification file set, a CORBA ORB parsing and a heterogeneous unmanned system hardware.

[0050] As a preferred embodiment, the IDL interface description specification file set is an extensible heterogeneous multi-UAV cooperative fire extinguishing framework. The typical scenarios described by the IDL interface description specification file set include: 1. UAV binding, unbinding and rebinding to the console. 2. The console setting task roles and task modes to the UAV. 3. The console sending search area coordinates and setting cruise speed to the cruise UAV. 4. The console controlling the UAV. 5. The console controlling the UAV to fly to a specified target point. 6. The UAV reporting its own state to the monitoring console. 7. The UAV reporting the reason for returning to the monitoring console when returning due to an event. 8. The console assigning idle fire extinguishing UAVs to the cruise UAV. 9. The UAV sending fire image, location information and fire extinguishing UAV preparation information to the monitoring console. 10. The console confirming or canceling the discovery of fire, fire location and preparation of the cruise UAV. 11. The cruise UAV controlling the movement and throwing of the fire extinguishing UAV.

[0051] As a preferred embodiment, the IDL interface description specification file set is parsed by the CORBA specification to generate a framework for the server and a stub for the client for different operating systems and programming languages in several components. The communication between components is responsible for the implementation conforming to the CORBA specification, and the network transmission problem does not need to be concerned when developing the collaborative scene. Since the framework describes the interoperation between unmanned systems at the task level, and a large amount of sensor data is not transmitted, the video stream is processed in the front end for identification. Therefore, the system proposed in the application can reduce the load of network bandwidth and is suitable for outdoor long-distance scenes.

[0052] As a preferred embodiment, the heterogeneous unmanned system hardware / heterogeneous unmanned aerial vehicle is at least configured to be able to externally connect a computing unit, so as to control the externally connectable computing unit of the unmanned aerial vehicle or the unmanned aerial vehicle dedicated ground control station. The interface specified in the IDL interface description specification file set is realized by the external computing unit.

[0053] As a preferred embodiment, taking the motion control of the unmanned aerial vehicle as an example: the computing platform of the unmanned aerial vehicle needs to determine what action to perform according to the incoming parameters, such as take-off, landing, unlocking, locking, etc. For different actions, the interfaces provided by the unmanned aerial vehicle are called. The console can control the motion of the unmanned aerial vehicle through the interfaces.

[0054] The above-mentioned heterogeneous multi-unmanned aerial vehicle cooperative fire extinguishing framework proposed in the application does not require the performance of the unmanned aerial vehicle, the algorithm of fire search path planning, the efficiency and accuracy of the fire identification and positioning algorithm. For unmanned aerial vehicles with more accurate and efficient algorithms, stronger performance, and more sensors, they can also quickly join the heterogeneous unmanned aerial vehicle fire extinguishing framework proposed in the application, thereby realizing the scalability of the framework.

[0055] The navigation of the unmanned aerial vehicle plays an important role in the cruise unmanned aerial vehicle fire search and the preliminary flight of the fire extinguishing unmanned aerial vehicle to the fire. As a preferred embodiment, the unmanned aerial vehicle proposed in the application realizes flight positioning and navigation based on the UWB (Ultra Wide Band) technology. Assuming that the two-dimensional coordinate system of the unmanned aerial vehicle on the ground is xy, the yaw (yaw) value provided by the IMU (Inertial Measurement Unit) sensor is 0 degrees as the starting angle when powered on, and left turning is positive. Let θ' be the angle returned by the IMU sensor of the unmanned aerial vehicle, θ x is the angle of the unmanned aerial vehicle with the x-axis of the coordinate system when powered on, and the first quadrant is positive. The position of the unmanned aerial vehicle is (X0, Y0), and the target point is (X1, Y1). The cruise speed of the unmanned aerial vehicle in the horizontal direction is V, and the speed components of the unmanned aerial vehicle are (V x , V y ), wherein V x is forward, and V yThe positive direction is right. The angle between the forward direction of the UAV target and the direction of the UAV body is θ. Then:

[0056] V x = V*cos(tan -1 ((Y1-Y0) / (X1-X0))-(θ'+θ x ));

[0057] V y =-V*sin(tan -1 ((Y1-Y0) / (X1-X0))-(θ'+θ x ))。

[0058] As a preferred embodiment, the UAV in the present application uses a training set taken in advance to train a deep convolutional network, and uses the deep convolutional network obtained after training to identify a video stream to determine whether there is a fire or other UAV in a specified range.

[0059] As a preferred embodiment, the UAV in the present application adopts a step-by-step positioning method to realize positioning of a fire. The cruise UAV preliminarily positions the fire under the condition of ignoring the distortion of the camera, instructs the fire extinguishing UAV to move to the fire occurrence location in the early stage of discovering the fire, and the cruise UAV further accurately positions the fire, thereby saving time.

[0060] Suppose that the cruise height of the cruise UAV is H, and the pitch angle (pitch angle) of the gimbal camera rotates 90 degrees downward. It is assumed that the actual length of the picture taken by the gimbal camera at the cruise height is R x meters, and the width is R y meters. C x and C y are the number of pixels of the length and width of the camera pixels, respectively. The UAV position is (X u , Y u ). θ' is the angle obtained by the IMU sensor. θ x is the angle between the X axis and the UAV when it is powered on. The center point of the identified fire is at the center of the picture and is set as the origin, and in the coordinate system with the X axis on the horizontal right side and the Y axis vertically upward, the pixel coordinates of the identified fire are (F x , F y ). The left upper corner of the picture is taken as the zero point. Under the condition of ignoring the distortion of the camera, the preliminary positioning of the fire can be calculated as (X f , Y f ):

[0061] X f =√((F y *(R y / C y ))2 + (F x * (R x / C x ) 2 ) * sin(tan -1 ((F y * (R y / C y )) / (F x * (R x / C x )) ) + θ' + θ x ) + X u ;

[0062] Y f = -√((F y * (R y / C y ) 2 + (F x * (R x / C x ) 2 ) * cos(tan -1 ((F y * (R y / C y )) / (F x * (R x / C x )) ) + θ' + θ x ) + Y u .

[0063] For accurate positioning, the unmanned aerial vehicle marks the center point of the fire in the picture according to the deep convolutional neural network, and adjusts the movement of the unmanned aerial vehicle according to the difference value data formed by the center point of the fire and the center point of the picture. When the difference value formed on the X axis and the difference value formed on the Y axis are all less than 10 pixels, it can be considered that the positioning is successful. At the same time, it can be assumed that the position of the unmanned aerial vehicle is the position of the fire.

[0064] When the cruise unmanned aerial vehicle is located directly above the fire, the fire extinguishing unmanned aerial vehicle is accurately positioned above the fire by the cruise unmanned aerial vehicle. The fire extinguishing unmanned aerial vehicle does not need to install a camera, and the identification of the cruise unmanned aerial vehicle can assist the fire extinguishing unmanned aerial vehicle to accurately throw the fire extinguishing bomb to the place where the fire is located. Let θ c be the angle obtained by the IMU sensor of the cruise unmanned aerial vehicle, θ c ' be the angle between the cruise unmanned aerial vehicle and the X axis when powered on. θ a be the angle obtained by the IMU sensor of the fire extinguishing unmanned aerial vehicle. θ a ' be the angle between the fire extinguishing unmanned aerial vehicle and the X axis when powered on. (V x1 , V y1) is the position of the fire-extinguishing unmanned aerial vehicle in the coordinate system set by the camera of the cruising unmanned aerial vehicle. Then the speed (V x2 , V y2 ) of the fire-extinguishing unmanned aerial vehicle in the coordinate system of the fire-extinguishing unmanned aerial vehicle is:

[0065] V x2 = V x1 *cos((θ c + θ c ')-(θ a + θ a '))-V y1 *sin((θ c + θ c ')-(θ a + θ a '));

[0066] V y2 = V x1 *sin((θ c + θ c ')-(θ a + θ a '))+V y1 *cos((θ c + θ c ')-(θ a + θ a '))。

[0067] The fire-extinguishing method proposed in the present application can include a search phase, a positioning phase, an auxiliary phase and a feedback phase executed in time sequence.

[0068] In the search phase, the operator sends the cruising range to the cruising unmanned aerial vehicle through the console. The cruising unmanned aerial vehicle searches according to the predetermined path. The operator sends the idle fire-extinguishing unmanned aerial vehicle information to the cruising unmanned aerial vehicle and waits for the cruising unmanned aerial vehicle to dispatch. The search path generation algorithm of the cruising unmanned aerial vehicle is extensible. The cruising unmanned aerial vehicle can use algorithms such as the plow algorithm, the updated probability density map, etc. to search the path.

[0069] In the search phase, the cruising unmanned aerial vehicle determines whether there is a fire in the video stream according to the pre-trained neural network model. When it is determined through the neural network that there is a fire, the cruising unmanned aerial vehicle calculates the preliminary position of the fire according to the pixel points in the obtained picture, and sends the fire image with labels, fire information and unmanned aerial vehicle position information to the monitor.

[0070] As a preferred embodiment, since there will always be inevitable errors in the calculation process of the neural network, the design of the human-in-the-loop is crucial. In the present application, the control console can request the operator to make a judgment feedback on the information returned by the UAV within a specified time through the display interface. The operator can inform the cruising UAV through the control console whether there is a fire in the cruising area. The UAV will save the photos fed back by the operator for further training to continuously improve the accuracy of identification. In the case that the operator confirms the existence of fire or does not confirm within the specified time, the cruising UAV sends the fire location calculated preliminarily to the fire extinguishing UAV, instructing the fire extinguishing UAV to fly to the approximate area where the fire is located and wait for further coordinated instructions. The cruising UAV enters the positioning stage.

[0071] Common precise positioning algorithms include methods such as flying to the top of the fire according to the video stream, multi-point positioning, infrared image laser image fusion positioning, etc. The algorithm for precise positioning is also scalable and upgradable. In order to facilitate subsequent provision of precise fire extinguishing assistance for the fire extinguishing UAV, it is preferably to complete positioning by flying to the top of the fire according to the video stream. After successful positioning, the cruising UAV sends positioning success information to the monitor and waits for the operator to confirm. In the case that the operator informs the cruising UAV of positioning failure through the control console, the cruising UAV repositions. In the case that the operator informs the cruising UAV of successful positioning through the control console or does not feed back confirmation within the specified time, the cruising UAV sends the precise fire location to the fire extinguishing UAV. The cruising UAV hovers and enters the assistance stage.

[0072] In the case that the cruising UAV enters the assistance stage, the cruising UAV will use the deep learning framework to identify other UAVs and use the aforementioned algorithm to control the fire extinguishing UAV through the control interface defined in IDL. Under the control of the cruising UAV, the fire extinguishing UAV is adjusted to the center of the picture or the center of the picture obtained by the camera of the cruising UAV. The cruising UAV sends a notification to the control console that it is ready to throw and waits for the operator to feed back confirmation.

[0073] In the assistance stage, in the case that the operator informs the cruising UAV to confirm throwing through the control console or does not feed back confirmation within the specified time, the cruising UAV controls the fire extinguishing UAV to throw down the fire extinguishing bomb. The fire extinguishing UAV reports the status to the control console and returns to the take-off point, automatically lands and waits to be refilled with fire extinguishing bombs. The cruising UAV enters the feedback stage.

[0074] In the feedback stage, the cruise unmanned aerial vehicle identifies the fire through the trained neural network. In the case of continuously not identifying the fire within a specified time, the cruise unmanned aerial vehicle sends the fire extinguishing success information to the console and waits for the operator to feedback the confirmation. In the case that the operator informs the cruise unmanned aerial vehicle through the console that the fire extinguishing is successful, the cruise unmanned aerial vehicle continues to cruise. In the case that the operator informs the cruise unmanned aerial vehicle through the console that the fire extinguishing is failed, the cruise unmanned aerial vehicle enters the positioning stage to restart.

[0075] The unmanned aerial vehicle plays a huge role in aerial photography. Due to the advantages of unmanned aerial vehicles in operation, and due to the popularity of various high-definition cameras, high-definition photos and high-definition videos of unmanned aerial vehicle aerial photography have been widely used. The popularity of high-definition photos and high-definition videos puts higher requirements on high-definition image and high-definition video processors. The space occupied by the image is getting larger and larger, which not only requires higher and higher processors and memories, but also due to the poor network environment of outdoor environment, the time required for transmitting high-definition pictures is longer, and the pressure of data transmission is also getting larger and larger. As a preferred embodiment, the unmanned aerial vehicle proposed in the application can be configured to save high-definition pictures in the local computer on board and transmit back compressed pictures through the network.

[0076] Since the calculation and analysis process of the processor and other components will inevitably have errors, the design of the human-in-the-loop is crucial. However, for the compressed pictures transmitted back through the network, the image resolution is not high, and the operator cannot quickly determine the fire condition through the transmitted images, that is, while reducing the data transmission pressure of the network, it will inevitably affect the judgment and analysis of the operator. If the operator fails to discover the fire and other hidden dangers in time due to image compression, it is easy to cause the operator to make a wrong judgment and even cause damage to personnel and equipment; even if the operator suspects that there is a fire hazard and requires the unmanned aerial vehicle to transmit back images with higher resolution, it also delays the timely judgment of the fire or the fire hazard in time.

[0077] As a preferred embodiment, the unmanned aerial vehicle at least comprises a data processing module, which is configured to convert the first fire image with the first resolution taken by the unmanned aerial vehicle at the first time into a second fire image with a second resolution lower than the first resolution and / or at least one third fire image with the first resolution, and transmit one or several of the second fire image, the corresponding identification area information and the at least one third fire image corresponding to the identification area information to the operator terminal at the second time. The identification area information can be recognized by the console of the operator terminal to perform image fusion and display of the information and / or the third fire image and the second fire image at the operator terminal. As a preferred embodiment, the unmanned aerial vehicle herein can mainly refer to a cruising unmanned aerial vehicle with a gimbal camera. The module in the application refers to a hardware, software or combined data processor capable of performing the relevant steps, and the data processing module can be a general term for several modules, that is, the execution steps corresponding to the data processing module can also be performed by several modules capable of information interaction with each other.

[0078] The first resolution refers to a high-definition image resolution parameter. As described above, the high-definition image has a large data transmission pressure, and therefore, the resolution thereof is adjusted to a relatively low second resolution at the unmanned aerial vehicle end before transmission, so as to greatly reduce the data transmission pressure without affecting the expression of the overall information such as the terrain; and the image recognition is also performed by preprocessing at the unmanned aerial vehicle end. The image processing module at the unmanned aerial vehicle end performs fire identification on the image while collecting the image, and stores the identification result as the identification area information. After the identification area information is transmitted to the operator terminal, the operator can quickly locate the fire area, or in other words, the operator can realize efficient observation with the aid of the system under low cognitive load.

[0079] When the operator receives the image returned from the unmanned aerial vehicle end, the area where there is a fire or suspected to have safety hazards will be magnified for observation. The system proposed in the prior art usually directly returns a complete image with high resolution. Although this is beneficial for the operator to selectively magnify the observation, the high-definition complete image puts a large pressure on data transmission, and the scaling of the complete high-definition image requires a high hardware configuration. When a large fire occurs, the hardware device at the operator end needs to continuously process a large number of high-definition images, which further increases the data processing pressure of the hardware device at the operator end, and is prone to problems such as lag or delay. To this end, the third fire image corresponding to the identification area information is extracted from the first fire image at the unmanned aerial vehicle end, and is transmitted to the operator end in a manner corresponding to the identification area information. The third fire image is a local first fire image with a first resolution. In this arrangement, the data processing pressure of directly returning a high-definition complete image is reduced, and at the same time, the operator is actively guided to observe the key area through the identification area information to improve the processing efficiency of the operator. Under this guidance, the operator is further ensured to quickly respond to the need to magnify the observation of the area where there is a fire or suspected to have safety hazards through the third fire image.

[0080] As a preferred embodiment, the data processing module sorts a plurality of identification area information by importance level and determines at least one identification area information according to the sorting result. The third fire image corresponding to the determined identification area information is extracted on the first fire image, and the data processing module transmits the extracted third fire image to the operator end together with the second fire image. Thus, the operator can quickly obtain the real scene image corresponding to the identification area information when following the identification area information for detail viewing. Through importance level sorting, the local area that needs to be viewed by the operator in the first time is screened out. Here, the third fire image determined by the sorting result can be one or two or more. The total amount of image data transmitted to the operator at the second time is small, while the rapid response of the system is ensured.

[0081] As a preferred embodiment, the data processing module can determine the number of identified area information to be determined from the sorting result according to the real-time network traffic and / or network communication quality. That is, in the case of large network traffic or poor network communication quality, the number of identified area information to be determined from the sorting result can be set to one, and accordingly the data processing module transmits only one third fire image in the data at the second time. In the case of small network traffic and good network communication quality, the number of identified area information to be determined from the sorting result can be set to two or three, and accordingly the data processing module transmits two or three third fire images in the data at the second time. As a preferred embodiment, a limitation condition is set in the data processing module, and the total data amount of all third fire images contained in the data transmitted by the data processing module at the second time should not exceed one half of the data amount of the first fire image with the first resolution.

[0082] As a preferred embodiment, the data processing module determines the second resolution based on the proportion of the identified area in the size of the first fire image. In other words, if the proportion of the identified area in the size of the first fire image exceeds a preset proportion threshold, the data processing module determines the first preset resolution parameter as the second resolution. If the proportion of the identified area in the size of the second fire image is lower than the preset proportion threshold, the data processing module determines the second preset resolution parameter as the second resolution. The first preset resolution parameter is lower than the second preset resolution parameter.

[0083] As a preferred embodiment, the proportion of the identified area in the size of the first fire image refers to the proportion of the sum of the image sizes of all third fire images corresponding to the identified area information in the image size of the first fire image.

[0084] For example, for a general close-up shot obtained first fire image, the identified area occupies a relatively large area in the first fire image, and even at a lower second resolution, it can be clearly determined whether there is a fire or other problems. For example, for a general long-range shot obtained first fire image, the identified area occupies a relatively small area in the first fire image, and if the second resolution is too low, it is not conducive to the operator to observe the terrain and other important information. For example, for a part of the long-range shot obtained first fire image, the identified area occupies a relatively large area in the first fire image due to the large fire, and it is difficult to determine the fire even if the visible light image is provided. In this case, the resolution of the second fire image can be determined at a lower second resolution, and at the same time, the infrared thermal image obtained by infrared imaging can be provided at a third resolution.

[0085] As a preferred embodiment, in the case that the second resolution is determined as the first preset resolution parameter, if the first fire image is in a close-up shooting mode, the data processing module is configured to convert the first fire image with the first resolution taken by the UAV at the first time into a second fire image with the second resolution lower than the first resolution, and transmit the second fire image and / or the corresponding identification area information to the operator terminal at the second time. The identification area information can be recognized by the console of the operator terminal to perform image fusion and display of the information and the second fire image at the operator terminal.

[0086] As a preferred embodiment, in the case that the second resolution is determined as the first preset resolution parameter, if the first fire image is in a close-up shooting mode, the data processing module is configured to convert the first fire image with the first resolution taken by the UAV at the first time into a second fire image with the second resolution lower than the first resolution, and transmit the second fire image and / or the corresponding identification area information to the operator terminal at the second time. The identification area information can be recognized by the console of the operator terminal to perform image fusion and display of the information and the second fire image at the operator terminal.

[0087] As a preferred embodiment, the image processing module transmits the images collected by it to the data processing module together with the shooting mode information used when collecting the images.

[0088] The identified region mentioned in the present application mainly refers to a local region where fire exists, may exist or has safety hazards obtained by the image processing module of the unmanned aerial vehicle after shooting and obtaining the image. The at least one identified region can be identified on the image floatingly according to the two-dimensional coordinates in the two-dimensional coordinate system defined by each image. The identified region information at least includes the two-dimensional coordinate information of the identified region in the two-dimensional coordinate system defined by the second fire image. As a preferred embodiment, different identified regions can be distinguished on the image according to the importance level of the sorting result of the identified region information. The distinguished identification mentioned here can refer to one or a combination of different color identification, different brightness color identification, different shape identification or different size shape identification.

[0089] As a preferred embodiment, the image fusion can be performed by the data processing module or the image processing module of the unmanned aerial vehicle, and then the unmanned aerial vehicle transmits the image containing the identified region information to the operator terminal. As a preferred embodiment, the image fusion can be performed by the console of the operator terminal, and the unmanned aerial vehicle transmits the image and the identified region information separately to the console. After obtaining the image, the unmanned aerial vehicle only needs to perform pre-processing such as compression and then transmit it to the console, and the console performs the image fusion and other data processing processes, which reduces the data processing pressure of the unmanned aerial vehicle terminal, improves the system response speed, and enables the operator to quickly understand the on-site situation.

[0090] Since the identified region is not a single point but a region with a certain area, the two-dimensional coordinate information corresponding to the identified region information should be a local region on the image defined by a certain range of horizontal coordinate values and a certain range of vertical coordinate values. That is, the identified region information corresponds to a first local limited range determined according to a certain regular shape. When the operator's input behavior operation falls within the first local limited range, the console calls the third fire image corresponding to the first local limited range and presents it to the display interface. The console displays the third fire image corresponding to the first local limited range on the fourth fire image with a certain magnification. That is, the operator can perform the first behavior operation related to the first identified region information on the fourth fire image obtained after image fusion through the display interface of the console. The console obtains the two-dimensional coordinates in the two-dimensional coordinate system defined by the fourth fire image corresponding to the first behavior operation at the third time when the first behavior operation is monitored, and determines that the operator inputs the first behavior operation related to the first identified region information when the two-dimensional coordinates fall within the at least one first local limited range. Then the console feeds back the third fire image that the operator needs to further view in high definition to the display interface for viewing.

[0091] As a preferred embodiment, the first behavior operation can be a mouse click or a behavior operation of the operator clicking the screen. Only in the case that the first behavior operation is related to the identification area information, the console responds to the first behavior operation to display the third fire image. By defining the operation behavior of the operator, the system can quickly and more accurately identify the operation behavior intention of the operator.

[0092] Since the fourth fire image displayed to the operator is only partially marked with the identification area information, the operator will usually check whether there is a fire or a safety hazard in other areas of the image that is not identified by the system after checking the identification area information. For this, if the operator clicks on the image in the prior art, the system cannot confirm the area range that the operator needs to view, and it usually displays the high-definition enlarged image in a certain area according to the default parameters, which is easy to cause a large deviation from the area range that the operator wants to view, and the operator may need to repeatedly click different positions in the area to completely view the real scene in the area. For this, in the present application, the console is configured to be able to identify the second behavior operation unrelated to the first identification area information that the operator inputs on the display interface to customize the area range. The console transmits the second behavior operation information to the unmanned aerial vehicle end at the fourth moment of monitoring the second behavior operation to retrieve at least one fifth fire image corresponding to the second behavior operation information from the unmanned aerial vehicle end according to the second local limited range customized by the operator. That is, the operator can customize the area range that he wants to view, and the console can retrieve and display the corresponding high-definition image according to the customized area range. Under this setting, the unmanned aerial vehicle end does not need to transmit and display the entire high-definition image, which reduces the pressure of data transmission, and the operator can customize to view the high-definition image of the specified area, which reduces the cognitive load of the operator.

[0093] As a preferred embodiment, the second behavior operation can be to define the area range that the operator wants to view by mouse or finger touch screen. The operator customizes the area range, so the area range obtained is usually irregular in shape.

[0094] As a preferred embodiment, the data processing module determines the fifth fire image from the first fire image based on the second local limited range customized by the operator.

[0095] As a preferred embodiment, in order to further reduce the pressure of system data transmission, the application segments the fifth fire image which needs to be transmitted in response to the second behavior operation information, saves bandwidth, and at the same time regulates the segmentation mode through a preset third local limited range, so that the image data under segmentation transmission can meet the viewing needs of the operator in time. The data processing module compares the second local limited range with the preset third local limited range. Unlike the second local limited range, the third local limited range is a parameter preset by the system, and the area range limited according to the third local limited range can better be viewed by the operator.

[0096] Since the operator often exceeds the area he wants to view when he autonomously delimits the area, and delimits a larger area, part of the area is not the area the operator wants to view, and the transmission of the high-definition data image of this part of the area becomes redundant, increasing the data transmission and data processing pressure. Especially for images collected by long-range shooting, compared with images collected by close-range shooting, if the operator delimits the same shape of the second local limited range, if the image delimited in the second local limited range is displayed with the same magnification, the image delimited and magnified in the long-range shooting may have the problem of insufficient magnification, requiring the operator to perform a secondary magnification operation to see the image of a smaller area. Conversely, the image delimited and magnified in the close-range shooting may have the problem of excessive magnification, requiring the operator to perform a secondary reduction operation to see a more complete image. In this regard, in the application, the data processing module determines the third local limited range corresponding to the shooting mode information at the same time of obtaining the shooting mode information. For example, the domain area of the third local limited range under the shooting mode of long-range shooting is smaller than the domain area of the third local limited range under the shooting mode of close-range shooting. For example, the magnification of the third local limited range under the shooting mode of long-range shooting is greater than the magnification of the third local limited range under the shooting mode of close-range shooting. That is, the third local limited range contains domain area information and magnification information. Especially for some existing systems that require the operator to adjust the image magnification by double-finger zooming, the double-finger flexibility and stability are poor, and often cannot stably achieve the expected magnification, and the closer to the expected magnification, the more likely it is to deviate more. Based on this, the third local limited range for the operator to view is adaptively set for different shooting modes, the image is displayed in a more suitable way for the operator to view, the problem that the operator needs to perform a secondary zooming operation after the high-definition image is displayed is improved, and the flexibility and stability of the prior art technical solution using double-finger zooming are improved, while the freedom of the operator's self-defined viewing area is retained.

[0097] To realize the segmented transmission of the third fire image, in the present application, in the case that the domain area of the second local limited range obtained by comparison exceeds the domain area of the third local limited range, the data processing module determines the two-dimensional coordinate information in the two-dimensional coordinate system defined by the first fire image and at least one fifth fire image with the first resolution corresponding to the two-dimensional coordinate information according to the second local limited range defined by the operator. The data processing module splits the fifth fire image into a first sub-image and a second sub-image according to the third local limited range, and transmits the sub-image corresponding to the third local limited range in the two sub-images to the operator terminal at the fifth time. The data processing module transmits the sub-image outside the third local limited range in the two sub-images to the operator terminal at the sixth time. The sub-image outside the third local limited range is obtained after the second local limited range is reduced by the third local limited range. The preset proportion threshold is a preset proportion value. The case that the exceeding proportion is greater than the preset proportion threshold means that the second local limited range exceeds the third local limited range by a large area. In this case, the third fire image is suitable for segmented transmission to save bandwidth.

[0098] In the case that the domain area of the second local limited range obtained by comparison exceeds the domain area of the third local limited range and the exceeding proportion is greater than the preset proportion threshold, the data processing module determines the sixth time in response to the third behavior operation of the operator inputting the sub-image transmitted to the operator terminal at the fifth time through the display interface. That is, the partial segmentation of the third fire image is to transmit another sub-image with relatively small data to the operator terminal when the operator has an explicit viewing demand, i.e. the operator views through the third behavior operation such as dragging. The bandwidth consumption is small. When the operator does not input the third behavior operation, there is no need to transmit other sub-image data, further reducing unnecessary redundant bandwidth consumption.

[0099] The fifth fire image is displayed on the fourth fire image but does not completely occupy the entire display interface, i.e. the fifth fire image is displayed as a window with a predetermined size and position. The size and position of the window mentioned here correspond to the preset third local limited range. Therefore, in the case that the domain area of the second local limited range exceeds the third local limited range, the operator can input the third behavior operation to drag or move the fifth fire image to display the edge area of the second local limited range defined by the operator.

[0100] In a case where the area of the second local limited range exceeds the area of the third local limited range and the exceeding ratio is not greater than the preset ratio threshold, the data processing module can determine a sixth time instant immediately following the fifth time instant according to the time sequence of the transmission. That is, different segments of the third fire image are transmitted to the operator terminal according to the time sequence. In this case, the area of the second local limited range exceeds the area of the third local limited range to a small extent, and the second local limited range can be directly transmitted to the operator terminal without the third action of the operator.

[0101] In a case where the area of the second local limited range is less than the area of the third local limited range, the data processing module extracts the fifth fire image with the first resolution from the first fire image according to the second local limited range based on the two-dimensional coordinate information. The data processing module transmits the fifth fire image to the operator terminal at the fifth time instant. The operator terminal processes the fifth fire image with the area of the third local limited range and the magnification ratio. Since the second local limited range is smaller than the third local limited range, the console performs the virtualization processing on the image-free area between the third local limited range and the second local limited range.

[0102] In another case, the operator generally uses a circle to divide a second local limited range, so that the second local limited range is basically a circle or an ellipse. In order to make the third local limited range correspond to the first fire image, the data processing module performs a center point query process of the second local limited range, and the center coordinate of the second local limited range is the center coordinate of the third local limited range, so that the third local limited range corresponds to the first fire image. The center point query process can be: constructing a virtual equilateral triangle in a local limited range similar to a ring or an ellipse, so that each vertex of the constructed virtual equilateral triangle is on the boundary line of the local limited range, three virtual high lines corresponding to the edges of the virtual equilateral triangle are established, and all the virtual high lines intersect at a virtual point, which is considered to be in the center of the local limited range.

[0103] It should be noted that the above-mentioned embodiments are examples, and those skilled in the art can think of various solutions under the inspiration of the disclosure of the present application, and these solutions also belong to the disclosed range of the present application and fall within the protection scope of the present application. Those skilled in the art should understand that the present application specification is illustrative and does not constitute a limitation on the claims. The protection scope of the present application is defined by the claims and their equivalents. The present application specification contains multiple inventive concepts, such as "preferably", "according to a preferred embodiment" or "optionally", which all indicate that the corresponding paragraph discloses an independent concept, and the applicant reserves the right to file a divisional application according to each inventive concept. Throughout the text, the features introduced by "preferably" are only optional ways, and should not be understood as necessarily set, therefore the applicant reserves the right to abandon or delete the relevant preferred features at any time.

Claims

1. A scalable heterogeneous multi-UAV cooperative fire extinguishing system, characterized in that, The system generates an IDL interface description specification file set according to the interface description class, and parses the IDL interface description specification file set through the CORBA specification to generate a framework for a server and a stub for a client in different operating systems and programming languages for the first component and the second component, wherein the cruise unmanned aerial vehicle as the first component calls the interface provided by the fire extinguishing unmanned aerial vehicle as the second component and controls the second component through the interface; the fire extinguishing unmanned aerial vehicle does not need to install a camera; In the search phase, when it is judged by the neural network that there is a fire, the cruise unmanned aerial vehicle calculates the preliminary position of the fire according to the pixel points in the obtained picture; and sends the fire image with labels, fire information and unmanned aerial vehicle position information to the monitoring console; in the case that the operator confirms that there is a fire or does not confirm within a specified time, the cruise unmanned aerial vehicle sends the preliminary calculated fire position to the fire extinguishing unmanned aerial vehicle, instructs the fire extinguishing unmanned aerial vehicle to fly to the approximate area where the fire is located and waits for further cooperative instructions; In the positioning phase, the cruise unmanned aerial vehicle completes positioning in the manner of flying to the top of the fire according to the video stream; in the case that the operator informs the cruise unmanned aerial vehicle of successful positioning through the console or does not feed back confirmation within a specified time, the cruise unmanned aerial vehicle sends the accurate fire position to the fire extinguishing unmanned aerial vehicle; In the auxiliary phase, the cruise unmanned aerial vehicle controls the fire extinguishing unmanned aerial vehicle through the control interface defined in the IDL; the cruise unmanned aerial vehicle sends notification information of preparation for throwing to the monitoring console, and in the case that the operator confirms the throwing through the console or does not feed back confirmation within a specified time, the cruise unmanned aerial vehicle controls the fire extinguishing unmanned aerial vehicle to throw down the fire extinguishing bomb; In the feedback phase, in the case that the fire is not continuously identified within a specified time, the cruise unmanned aerial vehicle sends fire extinguishing success information to the console and waits for the operator to feed back confirmation.

2. The system of claim 1, wherein, The components include at least one unmanned aerial vehicle configured to achieve flight positioning and navigation based on UWB technology, and the unmanned aerial vehicle calculates its speed component (V x , V y ) in a horizontal two-dimensional coordinate system according to the following formula: ； ， V is the cruising speed of the unmanned aerial vehicle in the horizontal direction; θ' is the angle returned by the IMU sensor of the unmanned aerial vehicle; θ x is the angle between the power-on unmanned aerial vehicle and the x-axis of the coordinate system; (X0, Y0) is the position of the unmanned aerial vehicle; (X1, Y1) is the position of the target point.

3. The system of claim 2, wherein, The unmanned aerial vehicle is configured to train a deep convolutional network using a training set taken in advance, and to identify the video stream using the deep convolutional network obtained after training to determine whether there is a fire or other unmanned aerial vehicle within a specified range.

4. The system of claim 3, wherein, The unmanned aerial vehicle at least includes a cruise unmanned aerial vehicle and a fire extinguishing unmanned aerial vehicle, the cruise unmanned aerial vehicle preliminarily locates the fire under the condition of ignoring the distortion of the camera, instructs the fire extinguishing unmanned aerial vehicle to move to the fire occurrence site in the early stage of discovering the fire, and the cruise unmanned aerial vehicle further accurately locates.

5. The system of claim 4, wherein, The cruise drone is configured to calculate preliminary positioning coordinates (X f , Y f ) of the fire according to the following formula under the condition of ignoring the distortion effect of the camera: X = (X1 - X0) / (X1 - X0) * (X2 - X0) + X0 Y = (Y1 - Y0) / (Y1 - Y0) * (Y2 - Y0) + Y0 ； ， wherein R x m is the actual length of the picture taken by the gimbal camera on the cruise UAV at cruising altitude, R y m is the actual width of the picture taken by the gimbal camera on the cruise UAV at cruising altitude; C x and C y pixel number of length and width of camera pixel, respectively; (X u , Y u ) are the positions of the UAVs; (F x , F y ) are pixel coordinates occupied by the identified fire.

6. The system of claim 5, wherein, The cruise unmanned aerial vehicle is configured to mark the center point of the fire in the picture according to the deep convolutional neural network, and adjust the movement of the fire extinguishing unmanned aerial vehicle according to the difference value data formed by the center point of the fire and the center point of the picture.

7. The system of claim 6, wherein, The cruise unmanned aerial vehicle is configured to mark the center point of the fire in the picture according to the deep convolutional neural network, and adjust the movement of the unmanned aerial vehicle according to the difference value data formed by the center point of the fire and the center point of the picture, and determine that the fire is located successfully when the difference value formed by the two in the X axis and the difference value formed in the Y axis are both less than a preset pixel.

8. The system of claim 7, wherein, The fire extinguishing unmanned aerial vehicle is configured to throw the fire extinguishing bomb to the fire location according to the identification result of the cruising unmanned aerial vehicle.

9. The system of claim 8, wherein, The fire extinguishing drone is configured to calculate its velocity component (V x2 , V y2 ) in a horizontal two-dimensional coordinate system according to the following formula: V = - (V x2 , V y2 ) · (V x2 , V y2 ) ； ； wherein θ c is the angle obtained by the IMU sensor of the cruise drone; θ a is the angle obtained by the IMU sensor on the fire extinguishing drone; θ a the angle between the unmanned fire extinguishing aircraft and the X axis when powered on (V x1 , V y1 ) is the location of the fire extinguishing UAV in the coordinate system set by the camera of the cruise UAV; θ c is the angle between the cruise UAV and the X-axis when powered on.

10. An extensible heterogeneous multi-UAV cooperative fire extinguishing system, comprising at least a UAV equipped with a gimbal camera and a console provided at an operator end, characterized in that, The unmanned aerial vehicle at least comprises a data processing module configured to: In the search phase, the first fire image with the first resolution obtained by the unmanned aerial vehicle at the first time is converted into a second fire image with a second resolution lower than the first resolution and at least one third fire image with the first resolution, and one or several of the second fire image, the identification area information corresponding to the second fire image and the at least one third fire image corresponding to the at least one identification area information are transmitted to the operator terminal at the second time, Wherein, the identification area information can be identified by the console of the operator terminal to fuse and display the information and / or the third fire image with the second fire image at the operator terminal; in the case that the operator confirms the existence of fire or does not confirm within a specified time, the cruising unmanned aerial vehicle sends the fire location calculated preliminarily to the fire extinguishing unmanned aerial vehicle; In the positioning phase, the cruising unmanned aerial vehicle completes positioning in the manner of flying to the fire directly above according to the video stream; in the case that the operator informs the cruising unmanned aerial vehicle of the successful positioning through the console or does not feed back the confirmation within a specified time, the cruising unmanned aerial vehicle sends the accurate fire location to the fire extinguishing unmanned aerial vehicle; In the auxiliary phase, the cruising unmanned aerial vehicle controls the fire extinguishing unmanned aerial vehicle through the control interface defined in the IDL; the cruising unmanned aerial vehicle sends the notification information of preparation for throwing to the monitoring console, and in the case that the operator informs the cruising unmanned aerial vehicle of the confirmation of throwing through the console or does not feed back the confirmation within a specified time, the cruising unmanned aerial vehicle controls the fire extinguishing unmanned aerial vehicle to throw down the fire extinguishing bomb; In the feedback phase, in the case that the fire is not identified continuously within a specified time, the cruising unmanned aerial vehicle sends the fire extinguishing success information to the console and waits for the feedback confirmation of the operator.

Images