Embedded unmanned aerial vehicle airborne multi-task control method and system

Abstract

The invention discloses an embedded unmanned aerial vehicle airborne multi-task control method and system, and relates to the technical field of image processing, and the embedded unmanned aerial vehicle airborne multi-task control method specifically comprises the steps: collecting a plurality of fire scene images through an unmanned aerial vehicle, and detecting the smoke information; generating a panoramic image based on the plurality of fire scene images; constructing a risk level map based on the panoramic image; and controlling the unmanned aerial vehicle to execute multiple tasks by using the risk level map. Through the above steps, the panoramic image of the fire scene can be generated, information such as the fire source position and the fire behavior severity can be comprehensively analyzed, and a command center is helped to rapidly master the global situation of the fire scene, dynamically distribute unmanned aerial vehicle tasks and cooperatively complete complex rescue tasks.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an embedded unmanned aerial vehicle (UAV) onboard multi-task control method and system, which realizes information sharing and collaborative operation capabilities among multiple UAVs, breaking the limitation of the prior art that UAVs operate independently. Background Art

[0002] With the rapid development of science and technology, drones, as a flexible, fast and efficient intelligent device, have been widely used in many fields, including agriculture, logistics, inspection, mapping, security and disaster relief. Especially in disaster relief, drones have become an important part of the modern disaster relief system with their flexible flight capabilities and the ability to carry a variety of sensors for real-time monitoring. At the disaster fire scene, drones can quickly reach the disaster area and conduct high-altitude reconnaissance, ground surveys, vital signs detection, communication relay and other tasks, significantly improving the efficiency of rescue. At the same time, drones do not require personnel to operate in dangerous areas, greatly reducing the risk of life safety of rescuers. Compared with traditional disaster relief methods, drones have the advantages of rapid deployment, real-time monitoring, and multi-task execution, especially in high-risk and complex disaster relief scenarios such as fires, earthquakes, and floods. They play an irreplaceable role.

[0003] However, although some progress has been made in the application of drones in fire rescue, the existing technology still has great limitations. The technology of drone fire rescue mainly relies on manual operation, that is, the pilot remotely controls the drone to complete the task. Each pilot can usually only control one drone, receive the images sent back by the drone through the control device and make decisions. However, there are many problems with this operation method. First, due to the danger of the fire scene, the pilot is often unable to get close to the fire scene and can only operate in a safe area far away from the fire scene. This remote control method limits the pilot's field of vision, and the information that can be mastered is only the images or data sent back by a single drone, lacking the ability to perceive the overall situation of the fire scene. The distribution of fire sources in the fire scene is complex and the fire spreads rapidly. The picture of a single perspective cannot meet the understanding of the overall picture of the fire scene, which may cause the pilot to miss key fire information, thereby affecting the scientific nature of the rescue decision. Secondly, in the existing technology, when multiple drones participate in fire rescue, they work independently and lack the ability to share information and collaborate. Each drone is operated by a different pilot, and the drones cannot exchange data in real time or collaborate to complete complex tasks. This "single-machine operation" mode not only increases the workload of pilots, but also easily leads to unreasonable task allocation. In addition, the existing fire rescue drone system is relatively simple in terms of fire scene monitoring, mainly limited to the monitoring of open flames, but there are still many deficiencies in the overall situation analysis of the fire scene. For example, the direction of smoke diffusion in the fire scene, the trend of fire spread, and the distribution of potential fire sources have not been effectively included in the monitoring scope. It is difficult to make a scientific judgment on the overall situation of the fire scene by relying solely on the monitoring of open flames. At the same time, when a fire occurs, there may be flammable areas that have not yet burned but have a high risk of burning, dry vegetation, etc. in the fire scene. The existing technology has a weak ability to identify these potential fire sources, and it is difficult to provide comprehensive fire scene information for ground rescue personnel. The limitations of this monitoring method directly lead to the lack of scientific deployment of rescue resources, and there may be waste of fire-fighting resources or incorrect deployment.

[0004] In view of the above problems, the present invention proposes an embedded UAV airborne multi-task control method and system, aiming to solve the deficiencies in the prior art. The system can integrate visual processing algorithms to achieve visual sharing and task collaboration of multiple UAVs. In a fire rescue scenario, the system can generate a panoramic image of the fire scene through images and data taken by multiple UAVs, and conduct a comprehensive analysis of information such as the location of the fire source and the severity of the fire, helping the command center to quickly grasp the overall situation of the fire scene, dynamically assign UAV tasks, and collaboratively complete complex rescue tasks. Summary of the invention

[0005] In view of this, the present invention provides an embedded UAV onboard multi-task control method, the method specifically comprises the following steps: S1: Use drones to collect multiple fire scene images and detect smoke information; S2: generating a panoramic image based on the multiple fire scene images; S3: constructing a risk level map based on the panoramic image; S4: Using the risk level map to control the drone to perform multiple tasks.

[0006] The present invention also provides an embedded unmanned aerial vehicle airborne multi-task control system, the system comprising: Image acquisition module: The image acquisition module uses a drone to collect multiple fire scene images and detect smoke information; Panoramic image generation module: the panoramic image generation module generates a panoramic image based on the multiple fire scene images; Risk level map generation module: the risk level map generation module constructs a risk level map based on the panoramic image; Mission control module: The mission control module uses the risk level map to control the drone to perform multiple tasks.

[0007] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned embedded unmanned aerial vehicle onboard multi-task control method when executing the computer program.

[0008] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned embedded unmanned aerial vehicle onboard multi-task control method is implemented.

[0009] Compared with the prior art, the present invention discloses an embedded UAV-mounted multi-task control method. The method is first based on using multiple UAVs to shoot multi-region images of the fire scene, and selects specific features based on the characteristics of the fire scene target to stitch the images to obtain a panoramic image. At the same time, the panoramic image is desmoked based on the multi-region image. In the defogging process, the present invention adopts a two-stage removal method to ensure that the panoramic image is as real and clear as possible, providing the necessary data basis for the subsequent risk level map establishment. Based on the panoramic image, the present invention uses the smoke area to track the smoke starting point and generate a weight map, guiding the key target detection of the smoke starting point, i.e., the open fire or potential ignition point, during the fire scene target detection, and uses the detection results to construct a risk level map to achieve multi-task control. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0011] Figure 1 It is a structural diagram of the embedded UAV airborne multi-task control method in this application; DETAILED DESCRIPTION

[0012] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0013] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0014] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspect described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0015] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the examples can be practiced without these specific details.

[0016] The technical solutions provided by various embodiments of the present application are described below in conjunction with the accompanying drawings.

[0017] The embodiment of this specification provides an embedded UAV onboard multi-task control method, which specifically includes the following steps: S1: Use drones to collect multiple fire scene images and detect smoke information; S2: generating a panoramic image based on the multiple fire scene images; S3: constructing a risk level map based on the panoramic image; S4: Using the risk level map to control the drone to perform multiple tasks.

[0018] Use drones to collect multiple fire scene images and detect smoke information: In a specific embodiment, the present invention uses a drone to first take high-altitude photos of the entire fire scene to quickly obtain the global location distribution and boundary information of the fire scene. On this basis, multiple drones are deployed to take low-altitude photos of local areas of the fire scene to ensure that high-resolution local image data is obtained and the entire fire scene is effectively covered.

[0019] First, the image acquisition of the fire scene starts with high-altitude photography. The goal of high-altitude photography is to quickly obtain the global scope and boundary information of the fire scene, and provide basic data support for subsequent low-altitude local photography tasks. Industrial-grade drones are preferred for high-altitude photography. The drone equipment should have strong wind resistance, long flight time, and high-resolution cameras, and be able to fly stably at high altitudes and clearly capture the scope of the fire scene. Preferably, DJI Matrice 300 RTK is used. The starting height of the drone high-altitude photography setting is The height setting needs to take into account the field of view of the drone camera and the maximum diameter of the estimated fire area. And the full coverage requirement of shooting. In a specific implementation, , where θ is the horizontal field of view of the drone camera lens. This formula can ensure that the drone covers the entire fire scene as much as possible in a single shot. If the calculated , the altitude is lower than the minimum safe flight altitude of the drone. For example, there may be thermal air currents or smoke above the fire scene. Adjust to the preset safety height, preferably 50 meters above the ground of the fire scene. In addition, when the fire scene is too large and a single drone cannot complete the full-scene shooting, a threshold height needs to be set and a regional shooting strategy needs to be adopted. The drone takes multiple shots in sequence at the threshold height according to the preset route. After the shooting is completed, the high-altitude image will be used to calibrate the overall scope of the fire scene and the zoning plan.

[0020] After the high-altitude fire scene images are collected, multiple drones are deployed to take low-altitude local images of the fire scene according to the fire scene range marked in the high-altitude images. The purpose of low-altitude shooting is to obtain detailed information about the fire scene and provide high-resolution local data for subsequent fire scene analysis and smoke detection. The fire scene range marked in the high-altitude image is divided into several small areas according to the grid, ensuring that the range of each small area is suitable for a single low-altitude shooting coverage by the drone. Each drone is responsible for shooting one or more grid areas. During the shooting process, it is necessary to ensure that the images of adjacent grid areas have overlapping parts to facilitate subsequent image stitching and fusion. The preferred overlap ratio is 20%-30%. Low-altitude shooting should have strong high temperature resistance, be able to fly stably in an environment close to the fire scene, and capture high-precision local images. The low-altitude shooting height The setting needs to comprehensively consider the fire situation, the safety of the drone and the resolution requirements of the local image. Preferably, the safety height of the low-altitude drone is set to a height of 20 meters above the ground of the fire scene.

[0021] During the low-altitude local image shooting process, the drone performs the task in hover shooting mode to ensure that the details of each grid area are covered. The image resolution must meet the standard of at least 4K. At the same time, the image format is uniformly saved in RAW format to facilitate subsequent image processing and stitching.

[0022] The present invention adopts a mode of collaborative operation of multiple drones, and each drone is equipped with an embedded system for receiving instructions from a ground control system (GCS) and performing shooting tasks. First, the ground control system is the core of the entire drone multi-task collaborative operation. GCS is mainly responsible for the analysis and processing of high-altitude images, the division of the fire area, task allocation, and real-time monitoring of the drone status. After the high-altitude image acquisition is completed, GCS divides the fire area into several independent grid areas through a built-in area segmentation algorithm, and each grid area represents a low-altitude shooting task. GCS will automatically allocate these grid areas to suitable drones according to the current state and task requirements of each drone, including position, power, and load. The logic of task allocation follows the following principle: priority is given to drones that are closest to the target area, have sufficient remaining power, and have a low current task load, thereby ensuring the efficient execution of the shooting task. Each drone is equipped with an embedded system, which is the core module for the drone to perform tasks, and has functions such as command reception, task execution, state feedback, and data interaction. The embedded system hardware includes a high-performance single-chip microcomputer or embedded processor as well as a communication module, a sensor module, and a data storage module. The software part includes task receiving module, task planning module, task execution module and status feedback module. When GCS sends task instructions to the UAV, the task receiving module of the embedded system will receive the task ID, coordinate range, shooting height and shooting parameters of the grid area. The task receiving module will pass this information to the task planning module. The task planning module automatically calculates the optimal flight path based on the current position, flight path and task requirements of the UAV. After shooting, the embedded system transmits the image data to GCS in real time through the communication module, and updates the task status to "completed". If a UAV fails to complete the task due to insufficient power, communication interruption or other failures during the task execution, the status feedback module of its embedded system will immediately feedback the progress and unfinished part of the current task to GCS. After receiving the feedback, GCS will reallocate the unfinished tasks to other UAVs to ensure the continuity and efficiency of the tasks. During the simultaneous shooting of multiple UAVs, the embedded system will strictly control the flight and shooting behavior according to the received task instructions. The captured image data will be annotated with metadata such as shooting time, shooting location and camera parameters in real time for subsequent image stitching and analysis. All image data taken by the drones are synchronously transmitted to the GCS through the communication module. The GCS will manage and store the data according to the mission ID and shooting location, and start the image stitching and processing process after the mission is completed.

[0023] After collecting multiple fire scene images, the smoke information in the fire scene images is first detected; There is a clear mapping relationship between the color of smoke at the fire scene and the burning materials. Different materials will generate smoke of specific colors due to their chemical composition and combustion characteristics when burning. When burning, organic materials such as wood and cotton fabrics usually release gray or light gray smoke. This is because these materials produce water vapor and trace carbon particles during the combustion process. When the combustion is more complete, the color of the smoke will be lighter. When plastics, rubber, oils and other synthetic materials are burned, due to incomplete combustion, a large amount of incompletely burned carbon particles, namely carbon black, will be released, thus forming black smoke or dark gray smoke. This phenomenon is particularly significant in high-density materials or when there is insufficient oxygen supply. Sulfur-containing materials or certain chemicals will release colored gases such as sulfur oxides (SO2) and nitrogen oxides when burning, causing the smoke to appear yellow-brown or light yellow. In addition, when highly volatile fuels such as alcohol, natural gas, and gasoline are burned, short-wave scattering of volatile substances at high temperatures can generate blue smoke. According to the above analysis, the present invention extracts the smoke information that may appear in the fire scene. Its purpose is not only to perform smoke removal processing in the panoramic image, but more importantly to provide necessary information guidance for the subsequent risk level map analysis and construction.

[0024] Specifically, detecting smoke information includes: performing color space conversion on the fire scene image; extracting the location and type of fire scene smoke based on color features, contrast features, and texture sparsity features; The color space conversion includes converting the fire scene image into YCbCr and HSV color spaces; the contrast feature is the brightness contrast in the window, and the contrast feature is defined as: ; in, Indicates the number of pixels in the window, represents the pixel set in the window, Represents the brightness value of the pixel in the window. Represents the average brightness in the window; The texture sparse feature is a texture feature extracted from a local binary pattern, and the texture coefficient is defined as: ; Among them, LBP is the local texture feature operator, is the image to be processed; White smoke in a fire scene has the optical characteristics of high brightness, low saturation, low contrast and sparse texture. Therefore, the white smoke detection conditions are: , , , ; in, , White smoke brightness minimum and maximum thresholds, is the saturation value of the pixel in the HSV color space, , , are respectively white smoke saturation threshold, contrast threshold, and sparse threshold. Preferably, =0.6, =0.9, , =10; Black smoke in a fire scene has the optical characteristics of low brightness, high saturation, high contrast and sparse texture. Therefore, the black smoke detection conditions are: , , , ; in, Black smoke has the highest brightness. is the saturation value of the pixel in the HSV color space, , , are respectively the smoke saturation threshold, contrast threshold, and sparse threshold. Preferably, =0.4, , =10; The yellow smoke in the fire scene has the optical characteristics of clear yellow-brown hue, medium brightness and sparseness. Therefore, the yellow smoke detection conditions are: , , ; in, , Yellow smoke hue minimum and maximum thresholds, , are yellow smoke saturation threshold and sparse threshold respectively. Preferably, =0.083, =0.139, =10.

[0025] S2: generating a panoramic image based on the multiple fire scene images; In the fire scene image panoramic stitching, in order to ensure the universality and robustness of the algorithm, the present invention selects universal features applicable to different fire scenes. The selected features are stable, not easily affected by dynamic factors such as flames and smoke, and widely exist in various fire scene environments.

[0026] The present invention selects geometric features as the most universal features in fire scenes. Straight line features are widely present in environments such as building walls, door frames, ground boundaries, tree trunks, etc. They have high stability, and the spatial distribution of straight lines can reflect the geometric structure of objects in the fire scene. Through the straight line detection algorithm, these straight line features can be extracted to provide a reliable basis for image alignment. At the same time, corner point features are a supplement to geometric features and are used to detect areas such as building corners and object edge intersections. These corner points also have good versatility and robustness in fire scenes and can be effectively extracted through corner point detection algorithms.

[0027] Secondly, the present invention also selects texture features as another important universal feature in the fire scene, which is widely present on the ground, such as soil, roads and object surfaces. The texture information has little dynamic change in the fire environment and can be extracted and matched through local feature descriptors. At the same time, the detailed information of the texture features provides local constraints for image stitching, especially in areas lacking obvious geometric features, showing strong adaptability. In addition, the present invention introduces contrast features for fire scene image stitching. Fixed objects in the fire scene usually have a higher contrast than dynamic objects such as flames and smoke. By utilizing the feature points in high-contrast areas, static features can be effectively distinguished from dynamic features, thereby improving the accuracy and stability of stitching.

[0028] Define the generated multiple fire scene images as , each image The position in the panoramic image grid is known, namely For the first image row and column coordinates in the grid; Specifically, generating a panoramic image includes: Initialize the panoramic image , construct the image set to be stitched ; splicing the images in the image set to be spliced ​​with the already spliced ​​panoramic image according to the grid positions, and updating the image set to be spliced ​​and the panoramic image at the same time; In each stitching, a straight line matching set of the image m and image p to be stitched is constructed , corner point matching And texture matching collection , and build a set of matching points ;Image stitching is achieved through image registration; Line matching set: ; Among them, the straight line feature set of image m is , is the number of straight lines, and the nth straight line is: , are the vertical distance from the line n to the origin and the angle with the horizontal axis, are the starting point and end point of the projection of line n along a certain direction respectively; Before each splicing, contrast screening is introduced to prevent the dynamic flame area from affecting the accuracy of feature point extraction. Specifically, the contrast map of the image m and image p to be spliced ​​is calculated, and the threshold is used for screening to eliminate the low-contrast area of ​​the image to be spliced; The image matching includes the identity matrix calculate: ; and coordinate transformation ,in, The projected image The pixel value of for The inverse matrix of .

[0029] After the panoramic image is generated, it is also necessary to perform smoke removal processing on the panoramic image. The specific process of smoke removal processing includes: Traverse each pixel position (x, y) in the panoramic image and construct the fire scene image set corresponding to the pixel position ,The fire images in the fire image set all cover the pixel positions (x, y) in the panoramic image; Deriving Panoramic Image Smoke Mask and clear mask : ; ; in, is the smoke mask of the t-th fire scene image, recording the smoke position information in the t-th fire scene image; Traverse each pixel position (x, y) in the panoramic image; when , that is, at least one fire scene image has no smoke at this location, then pixels are selected from the smoke-free image for replacement: ,in, at the same time, ; if , that is, if there are multiple fire scene images without smoke at this location, pixel fusion is used to further reduce noise; when 0, then global smoke processing is performed to determine the dominant smoke type , and determine the smoke removal algorithm based on the dominant smoke type; if white smoke is dominant, the smoke area is brightness smoothed, if black smoke is dominant, the smoke area is brightness enhanced, and if yellow smoke is dominant, the smoke area is median filtered; ; in, is the smoke type at the (x, y) position of t fire scene images, is the voting function.

[0030] The present invention adopts a two-stage smoke removal method during the smoke removal process, which can effectively improve the authenticity and clarity of the panoramic image, while taking into account the processing efficiency and the robustness of complex scenes. The core idea of ​​the first stage is to give priority to using smoke-free local images for pixel replacement. The greatest advantage of this method is that it directly retains the real information in the fire scene image, does not need to rely on complex smoke removal algorithms, and avoids the destruction of the details of the original image. By checking all local images covering a certain position of the panoramic image, if there is at least one local image without smoke at this position, the pixel value of the smoke-free area can be directly used to replace the pixel of the smoke area, ensuring the clarity and authenticity of the area. At the same time, when multiple local images are smoke-free at this position, further pixel fusion or the highest quality image pixels are selected to significantly reduce noise and improve the visual effect of the image. The present invention makes full use of the redundancy of multi-view data to ensure high-quality reconstruction of the panoramic image in the local smoke-free area.

[0031] Specifically, judging whether there is smoke at a certain location in the panoramic image depends on the comprehensive information of the smoke mask of the local image. If smoke is detected in all local images covering the location, it is reasonable to infer that the smoke at the location is a global or unavoidable smoke phenomenon; if there is at least one local image without smoke, it means that the smoke is a local phenomenon and can be replaced by an image without smoke. One-stage processing is more efficient and can significantly reduce the amount of calculation while retaining the details of the real scene.

[0032] For those areas where smoke exists in all local images, the unified smoke removal algorithm in the second stage can adopt targeted processing methods for different types of smoke, thereby achieving a global smoke removal effect. In summary, the smoke removal algorithm of the present invention can not only optimize the panoramic image by preferentially using the real image, but also make up for the lack of local data through the algorithm when necessary.

[0033] based on Collect smoke sequence image set: ,in, is the s-th frame image of the q-th smoke area, Indicates the number of captured frames; The specific acquisition process of the smoke sequence image set includes: calibrating the smoke center point and circumscribed moment in the panoramic image to obtain the smoke region set: ;in,( ) is the center point of the smoke area, are the horizontal and vertical coordinates of the maximum and minimum values ​​of the circumscribed moment, is the number of smoke areas; The ground control system plans a flight path for the drone based on the center point of the smoke area, sorts the smoke areas by size, and gives priority to shooting the larger smoke areas; the drone arrives at the center point of the smoke area according to the planned flight path and collects the corresponding smoke sequence images: .

[0034] According to the smoke sequence image set, the optical flow method is used to track the smoke starting point, and the first weight map of the panoramic image based on Gaussian distribution is constructed based on all the smoke starting points; The panoramic image is fused with the first weighted image to obtain a regional enhanced panoramic image, and multi-target recognition is performed on the regional enhanced panoramic image; The target recognition network includes a backbone network, a multi-scale fusion module, a feature enhancement module and a detection head. The backbone network obtains 4 multi-scale feature maps ; The multi-scale fusion module includes an edge feature module, an edge enhancement module and an edge fusion module, The edge feature module is defined as: ; ; in, and Represents edge feature map and mask edge feature map, Conv represents convolution operation, and the subscript is the convolution kernel size; The multi-scale fusion module is defined as:

[0035] in, , is the feature map of the i-th fusion module, and Represent pixel-by-pixel multiplication and addition respectively; ; ; in, It is the feature map of the first fusion module; The feature enhancement module is defined as: ; ; ; ; ; ; in, , are the two inputs of the enhancement module, To enhance the output of the module, They are shallow feature maps Four sub-feature maps divided by channel, They are the intermediate feature maps of the feature enhancement module; The feature enhancement module has three sets of feature maps. The first set of inputs is: , , the second set of input: , Output feature map obtained by processing the first set of inputs for the feature enhancement module, the third set of inputs: , output feature maps obtained by processing the second set of inputs for the feature enhancement module; The three sets of output feature maps of the feature enhancement module are input to the detection head to obtain the target recognition results of the panoramic image; The target recognition results include personnel targets, open flame targets, and plant targets. When there is no open flame target in the smoke starting point area, the area is marked as a potential fire source target.

[0036] Constructing a risk level map based on the panoramic image specifically includes: Construct an initial risk level matrix based on the target recognition results; calculate the mutual influence factors based on the spatial relationship between targets, update the initial risk level matrix, and obtain a risk level map Each value in the initial risk level matrix is ​​a risk value corresponding to the target covered by the pixel position; The mutual influence factors include the diffusion factor between the open fire target and the plant target, the potential combustion factor between the potential fire source target and the plant target, and the safety distance factor between the personnel target and the open fire target. The above mutual influence factors use Gaussian kernel function to map the relationship between the distance between the targets and the danger value.

[0037] Using the risk level map to control the drone to perform multiple tasks As the control center, GCS is responsible for assigning tasks to drones based on the risk level map, the real-time status of drones (and task priorities). GCS prioritizes the assignment of drones to firefighting and personnel evacuation tasks based on the location of high-risk areas in the risk level map, and prioritizes the dispatch of drones closest to the target task to reduce task response time. When the risk level in a certain area increases, GCS dispatches drones in the surrounding areas to collaborate and complete emergency tasks first.

[0038] The present invention also provides an embedded unmanned aerial vehicle airborne multi-task control system, the system comprising: Image acquisition module: The image acquisition module uses a drone to collect multiple fire scene images and detect smoke information; Panoramic image generation module: the panoramic image generation module generates a panoramic image based on the multiple fire scene images; Risk level map generation module: the risk level map generation module constructs a risk level map based on the panoramic image; Mission control module: The mission control module uses the risk level map to control the drone to perform multiple tasks.

[0039] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned embedded unmanned aerial vehicle onboard multi-task control method when executing the computer program.

[0040] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned embedded unmanned aerial vehicle onboard multi-task control method is implemented.

[0041] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0042] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An embedded UAV airborne multi-task control method, characterized in that: The control method comprises the following steps: S1: Use drones to collect multiple fire scene images and detect smoke information; S2: generating a panoramic image based on the multiple fire scene images; S3: constructing a risk level map based on the panoramic image; S4: Using the risk level map to control the drone to perform multiple tasks.

2. The method for controlling multi-tasks on board an embedded UAV according to claim 1, characterized in that: Detecting smoke information includes: performing color space conversion on the fire scene image; extracting the location and type of fire scene smoke based on color features, contrast features, and texture sparsity features.

3. The embedded UAV airborne multi-task control method according to claim 1, characterized in that: Generating a panoramic image specifically includes: initializing the panoramic image , construct the image set to be stitched ; splicing the images in the image set to be spliced ​​with the already spliced ​​panoramic image according to the grid positions, and updating the image set to be spliced ​​and the panoramic image at the same time; In each stitching, a straight line matching set of the image m and image p to be stitched is constructed , corner point matching And texture matching collection , and build a set of matching points ;Image stitching is achieved through image registration; Line matching set: ; Among them, the straight line feature set of image m is , is the number of straight lines, and the nth straight line is: , are the vertical distance from the line n to the origin and the angle with the horizontal axis, are the starting point and end point of the projection of line n along a certain direction respectively.

4. The multi-task control method based on embedded UAV according to claim 3 is characterized in that: After the panoramic image is generated, it is also necessary to perform smoke removal processing on the panoramic image. The specific process of smoke removal processing includes: Traverse each pixel position (x, y) in the panoramic image, and construct a fire scene image set corresponding to the pixel position. The fire scene images in the fire scene image set all cover the pixel position (x, y) in the panoramic image. Deriving Panoramic Image Smoke Mask and clear mask : ; ； in, is the smoke mask of the t-th fire scene image, recording the smoke position information in the t-th fire scene image.

5. The method for multi-task control based on embedded UAV according to claim 4 is characterized in that: Calibrate the smoke center point and circumscribed moment in the panoramic image to obtain the smoke area set: ;in, ( ) is the center point of the smoke area, are the horizontal and vertical coordinates of the maximum and minimum values ​​of the circumscribed moment, is the number of smoke areas; The ground control system plans a flight path for the drone based on the center point of the smoke area. The drone follows the planned flight path to the center point of the smoke area and collects the corresponding smoke sequence images: .

6. The method for multi-task control based on embedded UAV according to claim 1, characterized in that: According to the smoke sequence image set, the optical flow method is used to track the smoke starting point, and a first weight map of the panoramic image based on Gaussian distribution is constructed based on all the smoke starting points; the panoramic image is fused with the first weight image to obtain a regional enhanced panoramic image, and multi-target recognition is performed on the regional enhanced panoramic image.

7. The multi-task control method based on embedded UAV according to claim 1, characterized in that: Construct an initial risk level matrix based on the target recognition results; calculate the mutual influence factors based on the spatial relationship between targets, update the initial risk level matrix, and obtain a risk level map; Each value in the initial risk level matrix is ​​a risk value corresponding to the target covered by the pixel position.

8. An embedded UAV airborne multi-task control system, characterized in that: The system includes: Image acquisition module: The image acquisition module uses a drone to collect multiple fire scene images and detect smoke information; Panoramic image generation module: the panoramic image generation module generates a panoramic image based on the multiple fire scene images; Risk level map generation module: the risk level map generation module constructs a risk level map based on the panoramic image; Mission control module: The mission control module uses the risk level map to control the drone to perform multiple tasks.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for controlling multi-tasks on board an embedded unmanned aerial vehicle is implemented.

10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements an embedded unmanned aerial vehicle onboard multi-task control method as claimed in any one of claims 1 to 7.

Images