An Embedded UAV Onboard Multi-Mission Control Method and System

Abstract

The present application discloses an embedded UAV airborne multi-task control method and system, which relates to the technical field of image processing. The specific steps of the embedded UAV airborne multi-task control method of the present invention are as follows: collecting multiple fire scene images by the UAV and detecting smoke information; generating a panoramic image based on the multiple fire scene images; constructing a risk level map based on the panoramic image; and controlling the UAV to execute multi-tasks by using the risk level map. Through the above steps, a panoramic image of the fire scene can be generated, and information such as the location of the fire source and the severity of the fire can be comprehensively analyzed, helping the command center quickly grasp the overall situation of the fire scene, dynamically allocate UAV tasks, and cooperate to complete complex rescue tasks.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and particularly to an embedded UAV airborne multi-task control method and system. This method realizes information sharing and collaborative operation capabilities among multiple UAVs, breaking the limitations of the existing technology where UAVs operate independently. Background Art

[0002] With the rapid development of technology, as a flexible, fast, and efficient intelligent device, UAVs have been widely used in multiple fields, including agriculture, logistics, inspection, surveying and mapping, security, and disaster relief. Especially in disaster relief, UAVs, with their flexible flight capabilities and the characteristics of carrying multiple sensors for real-time monitoring, have become an important part of the modern disaster relief system. In the disaster fire scene, UAVs can quickly reach the affected area to perform tasks such as high-altitude reconnaissance, ground survey, vital sign detection, and communication relay, significantly improving the rescue efficiency. At the same time, UAVs do not require personnel to operate in dangerous areas, greatly reducing the life safety risk of rescue personnel. Compared with traditional disaster relief means, UAVs have advantages such as rapid deployment, real-time monitoring, and multi-task execution, and have shown an irreplaceable role especially in high-risk and complex disaster relief scenarios such as fires, earthquakes, and floods.

[0003] However, although some progress has been made in the application of drones in fire rescue, the existing technology still has great limitations. The technology of drone fire rescue mainly relies on manual operation, that is, the pilot remotely controls the drone to complete the task. Each pilot can usually only control one drone, receive the images sent back by the drone through the control device and make decisions. However, there are many problems with this operation method. First, due to the danger of the fire scene, the pilot is often unable to get close to the fire scene and can only operate in a safe area far away from the fire scene. This remote control method limits the pilot's field of vision, and the information that can be mastered is only the images or data sent back by a single drone, lacking the ability to perceive the overall situation of the fire scene. The distribution of fire sources in the fire scene is complex and the fire spreads rapidly. The picture of a single perspective cannot meet the understanding of the overall picture of the fire scene, which may cause the pilot to miss key fire information, thereby affecting the scientific nature of the rescue decision. Secondly, in the existing technology, when multiple drones participate in fire rescue, they work independently and lack the ability to share information and collaborate. Each drone is operated by a different pilot, and the drones cannot exchange data in real time or collaborate to complete complex tasks. This "single-machine operation" mode not only increases the workload of pilots, but also easily leads to unreasonable task allocation. In addition, the existing fire rescue drone system is relatively simple in terms of fire scene monitoring, mainly limited to the monitoring of open flames, but there are still many deficiencies in the overall situation analysis of the fire scene. For example, the direction of smoke diffusion in the fire scene, the trend of fire spread, and the distribution of potential fire sources have not been effectively included in the monitoring scope. It is difficult to make a scientific judgment on the overall situation of the fire scene by relying solely on the monitoring of open flames. At the same time, when a fire occurs, there may be flammable areas that have not yet burned but have a high risk of burning, dry vegetation, etc. in the fire scene. The existing technology has a weak ability to identify these potential fire sources, and it is difficult to provide comprehensive fire scene information for ground rescue personnel. The limitations of this monitoring method directly lead to the lack of scientific deployment of rescue resources, and there may be waste of fire-fighting resources or incorrect deployment.

[0004] In view of the above problems, the present invention proposes an embedded UAV airborne multi-task control method and system, aiming to solve the deficiencies in the prior art. The system can integrate visual processing algorithms to achieve visual sharing and task collaboration of multiple UAVs. In a fire rescue scenario, the system can generate a panoramic image of the fire scene through images and data taken by multiple UAVs, and conduct a comprehensive analysis of information such as the location of the fire source and the severity of the fire, helping the command center to quickly grasp the overall situation of the fire scene, dynamically assign UAV tasks, and collaboratively complete complex rescue tasks. Summary of the invention

[0005] In view of this, the present invention provides an embedded UAV onboard multi-task control method, the method specifically comprises the following steps:

[0006] S1: Use a drone to collect multiple fire scene images and detect smoke information;

[0007] S2: Generate a panoramic image based on the multiple fire scene images;

[0008] S3: Construct a risk level map based on the panoramic image;

[0009] S4: Use the risk level map to control the drone to perform multiple tasks.

[0010] The present invention also provides an embedded drone airborne multi - task control system, which includes:

[0011] Image acquisition module: The image acquisition module uses a drone to collect multiple fire scene images and detect smoke information;

[0012] Panoramic image generation module: The panoramic image generation module generates a panoramic image based on the multiple fire scene images;

[0013] Risk level map generation module: The risk level map generation module constructs a risk level map based on the panoramic image;

[0014] Task control module: The task control module uses the risk level map to control the drone to perform multiple tasks.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above - mentioned embedded drone airborne multi - task control method is implemented.

[0016] The present invention also provides a computer - readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above - mentioned embedded drone airborne multi - task control method is implemented.

[0017] Compared with the prior art, the present invention discloses an embedded drone airborne multi - task control method. This method first takes multiple images of multiple areas of a fire scene using multiple drones, selects specific features based on the characteristics of fire scene targets for image stitching to obtain a panoramic image, and at the same time removes smoke from the panoramic image based on the multiple - area images. In the de - fogging process, the present invention adopts a two - stage removal method to ensure that the panoramic image is as real and clear as possible, providing a necessary data basis for the subsequent establishment of the risk level map. Based on the panoramic image, the present invention uses the smoke area to track the starting point of the smoke and generates a weight map, guiding the key target detection of the starting point of the smoke, that is, the open fire or potential ignition point, during the fire scene target detection, and uses the detection results to construct a risk level map to achieve multi - task control. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] To more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings required for use in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0019] Figure 1 is the structural diagram of the embedded UAV airborne multi - task control method in the present application; Specific embodiments

[0020] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0021] The following uses specific examples to illustrate the implementation manners of the present application. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The present application can also be implemented or applied through other different specific implementation manners. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the scope of protection of the present application.

[0022] It should be noted that the following describes various aspects of the embodiments within the scope of the appended claims. It should be obvious that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is illustrative only. Based on the present application, those skilled in the art should understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspects described herein can be used to implement the device and / or practice the method. In addition, this device and / or this method can be implemented using other structures and / or functions in addition to one or more of the aspects described herein.

[0023] In addition, in the following description, specific details are provided for a thorough understanding of the examples. However, those skilled in the art will understand that the examples can be practiced without these specific details.

[0024] The following, with reference to the accompanying drawings, illustrates the technical solutions provided by the embodiments of the present application.

[0025] The embodiments of this specification provide an embedded UAV airborne multi - task control method, and this method specifically includes the following steps:

[0026] S1: Use a drone to collect multiple fire scene images and detect smoke information;

[0027] S2: Generate a panoramic image based on the multiple fire scene images;

[0028] S3: Construct a risk level map based on the panoramic image;

[0029] S4: Use the risk level map to control the drone to perform multiple tasks.

[0030] Use a drone to collect multiple fire scene images and detect smoke information:

[0031] In a specific embodiment, the present invention first uses a drone to take an aerial photo of the overall scope of the fire scene to quickly obtain the global position distribution and boundary information of the fire scene. On this basis, multiple drones are deployed to take low-altitude photos of the local areas of the fire scene to ensure obtaining high-resolution local image data and effectively covering the entire fire scene scope.

[0032] First, the image collection of the fire scene starts with an aerial photo. The goal of the aerial photo is to quickly obtain the global scope and boundary information of the fire scene to provide basic data support for subsequent low-altitude local photo tasks. Industrial-grade drones are preferably used for aerial photos. The drone equipment should have strong wind resistance, long battery life, and a high-resolution camera, and be able to fly stably at high altitude and clearly capture the fire scene scope. Preferably, DJI Matrice 300RTK is used. The starting height for the drone's aerial photo is set to Height1. The setting of this height needs to comprehensively consider the field of view angle of the drone camera, the estimated maximum diameter Diameter of the fire scene scope, and the full-coverage requirement for shooting. In a specific embodiment, where θ is the horizontal field of view angle of the drone camera lens. Through this formula, it can be ensured that the drone can cover the entire fire scene scope as much as possible in a single shot. If the calculated Height1 is lower than the minimum safe flight height of the drone, for example, there may be hot air currents or smoke above the fire scene, then Height1 needs to be adjusted to the preset safe height, preferably a height 50 meters above the fire scene ground. In addition, when the fire scene scope is too large and a single drone cannot complete the full-field shooting, a threshold height needs to be set, and a sub-region shooting strategy is adopted. The drone takes multiple shots in sequence along the preset flight path at the threshold height. After the shooting is completed, the aerial images will be used to calibrate the overall scope and sub-region planning of the fire scene.

[0033] After the high-altitude fire scene images are collected, according to the fire scene range calibrated in the high-altitude images, multiple drones are deployed to take low-altitude local images of the fire scene. The purpose of low-altitude shooting is to obtain the detailed information of the fire scene and provide high-resolution local data for subsequent fire scene analysis and smoke detection. The fire scene range calibrated in the high-altitude images is divided into several small areas according to the grid, ensuring that the range of each small area is suitable for single low-altitude shooting coverage by the drone. Each drone is responsible for shooting one or more grid areas, and during the shooting process, it is necessary to ensure that the images of adjacent grid areas have overlapping parts for subsequent image stitching and fusion. The preferred overlapping ratio is 20%-30%. The low-altitude shooting should have strong high-temperature resistance, be able to fly stably in an environment close to the fire scene, and capture high-precision local images. The setting of the low-altitude shooting height Height2 needs to comprehensively consider the fire situation of the fire scene, the safety of the drone, and the resolution requirements of the local images. Height2 < Height1. Preferably, the safe height of the low-altitude drone is set at a height 20 meters above the ground of the fire scene.

[0034] During the process of low-altitude local image shooting, the drone hovers shooting mode is used to execute the task to ensure the coverage of the details of each grid area. The resolution of the image needs to meet at least the 4K standard. At the same time, the format of the image is uniformly saved as the RAW format for subsequent image processing and stitching.

[0035] The present invention adopts a mode of multi - drone collaborative operation. Each drone is equipped with an embedded system, which is used to receive instructions from the Ground Control System (GCS) and execute the shooting task. First of all, the ground control system is the core of the multi - task collaborative operation of the entire drone. The GCS is mainly responsible for the analysis and processing of high - altitude images, the division of the fire area, task allocation, and the real - time monitoring of the drone status. After the high - altitude image acquisition is completed, the GCS divides the fire area into several independent grid regions through the built - in region segmentation algorithm. Each grid region represents a low - altitude shooting task. The GCS will automatically allocate these grid regions to suitable drones according to the current status and task requirements of each drone, including position, power, and payload. The logic of task allocation follows the following principle: give priority to the drone that is closest to the target area, has sufficient remaining power, and has a low current task load, so as to ensure the efficient execution of the shooting task. Each drone is equipped with an embedded system, which is the core module for the drone to execute tasks and has functions such as instruction reception, task execution, status feedback, and data interaction. The hardware of the embedded system includes a high - performance single - chip microcomputer or an embedded processor, as well as a communication module, a sensor module, and a data storage module. The software part includes a task reception module, a task planning module, a task execution module, and a status feedback module. When the GCS sends a task instruction to the drone, the task reception module of the embedded system will receive the task ID, coordinate range, shooting altitude, and shooting parameters of the grid region. The task reception module will transfer this information to the task planning module. The task planning module combines the current position, flight path, and task requirements of the drone to automatically calculate the optimal flight path. After the shooting is completed, the embedded system transmits the image data to the GCS in real time through the communication module, and at the same time updates the task status to "completed". If a drone fails to complete the task due to insufficient power, communication interruption, or other failures during the task execution, the status feedback module of its embedded system will immediately feedback the progress and uncompleted part of the current task to the GCS. After receiving the feedback, the GCS will re - allocate the uncompleted task to other drones to ensure the continuity and efficiency of the task. During the simultaneous shooting of multiple drones, the embedded system will strictly control the flight and shooting behaviors according to the received task instructions. The captured image data will be marked with meta - information such as shooting time, shooting position, and camera parameters in real time for subsequent image stitching and analysis. The image data captured by all drones is synchronously transmitted to the GCS through the communication module. The GCS will manage and store the data according to the task ID and shooting position, and start the image stitching and processing process after the task is completed.

[0036] After collecting multiple fire images, first detect the smoke information in the fire images;

[0037] There is a clear mapping relationship between the smoke color at the fire scene and the burning materials. Different materials will generate smoke of specific colors when burning due to their chemical compositions and combustion characteristics. Organic materials such as wood and cotton fabrics usually release grayish-white or light gray smoke when burning. This is because water vapor and trace amounts of carbon particles are produced during the combustion process of these materials. When the combustion is relatively complete, the smoke color will be lighter. When plastics, rubbers, oils, and other synthetic materials burn, due to incomplete combustion, a large amount of unburned carbon particles, that is, carbon black, will be released, thus forming black smoke or dark gray smoke. This phenomenon is particularly significant in the case of high-density materials or insufficient oxygen supply. Sulfur-containing materials or certain chemicals will release colored gases such as sulfur oxides (SO2) and nitrogen oxides when burning, resulting in yellowish-brown or light yellow smoke. In addition, highly volatile fuels such as alcohol, natural gas, and gasoline can generate blue smoke due to the short-wave scattering phenomenon of volatile substances at high temperatures when burning. According to the above analysis, the present invention extracts the possible smoke information at the fire scene. The purpose is not only to perform de-smoking processing in the panoramic image, but more importantly, to provide necessary information guidance for the subsequent construction of the risk level map analysis.

[0038] Specifically, the detection of smoke information includes: performing color space conversion on the fire scene image; extracting the position and type of fire scene smoke based on color features, contrast features, and texture sparsity features.

[0039] Among them, the color space conversion includes converting the fire scene image to the YCbCr and HSV color spaces; the contrast feature is the brightness contrast in the window, and the contrast feature is defined as:

[0040]

[0041] Among them, Ω represents the number of pixels in the window, Ω(x, y) represents the pixel set in the window, and Y(i, j) represents the brightness value of the pixel point in the window. represents the average brightness in the window;

[0042] The texture sparsity feature is the texture feature extracted by the local binary pattern, and the texture sparsity is defined as:

[0043]

[0044] Among them, LBP is the local texture feature operator, and Image(i, j) is the image to be processed.

[0045] The white smoke in the fire scene has the optical characteristics of high brightness, low saturation, low contrast, and texture sparsity. Therefore, the white smoke detection conditions are:

[0046] Among them, The minimum and maximum thresholds of the white smoke brightness, and S(x,y) is the saturation value of the pixel in the HSV color space. They are respectively the saturation threshold, contrast threshold, and sparsity threshold of the white smoke. Preferably,

[0047] The black smoke in the fire scene has the optical characteristics of low brightness, high saturation, high contrast, and sparse texture. Therefore, the black smoke detection conditions are:

[0048] Among them, The black smoke has the maximum brightness, and S(x,y) is the saturation value of the pixel in the HSV color space. They are respectively the saturation threshold, contrast threshold, and sparsity threshold of the black smoke. Preferably,

[0049] The yellow smoke in the fire scene has the optical characteristics of a distinct yellow-brown hue, medium brightness, and sparsity. Therefore, the yellow smoke detection conditions are:

[0050] Among them, The minimum and maximum thresholds of the yellow smoke hue, They are respectively the saturation threshold and sparsity threshold of the yellow smoke. Preferably,

[0051] S2: Generate a panoramic image based on the multiple fire scene images;

[0052] In the panoramic stitching of fire scene images, in order to ensure the generality and robustness of the algorithm, the present invention selects general features applicable to different fire scenes. The selected features are stable, not easily affected by dynamic factors such as flames and smoke, and are widely present in various fire scene environments.

[0053] The present invention selects geometric features as the most general features in the fire scene. Straight line features are widely present in environments such as building walls, door frames, ground boundaries, and tree trunks. They have high stability, and the spatial distribution of the straight lines can reflect the geometric structure of the objects in the fire scene. Through the straight line detection algorithm, these straight line features can be extracted to provide a reliable basis for image alignment. At the same time, corner point features are a supplement to geometric features, used to detect areas such as building corners and intersections of object edges. These corner points also have good generality and robustness in the fire scene and can be effectively extracted through the corner point detection algorithm.

[0054] Secondly, the present invention also selects texture features as another important general feature in the fire scene, which widely exist on the ground, such as soil, roads, and the surfaces of objects. The texture information has relatively small dynamic changes in the fire scene environment and can be extracted and matched through local feature descriptors. At the same time, the detailed information of the texture features provides local constraints for image stitching, especially showing strong adaptability in areas lacking obvious geometric features. In addition, the present invention introduces contrast features for fire scene image stitching. Fixed objects in the fire scene usually have higher contrast compared to dynamic things such as flames and smoke. By using the feature points in high-contrast areas, static features and dynamic features can be effectively distinguished, thereby improving the accuracy and stability of stitching.

[0055] Define the generated multiple fire scene images as {I1, I2, I3, ……, I N}, and the position of each image I k in the panoramic image grid is known, that is, (r k , c k ) is the row and column coordinates of the image I k in the grid;

[0056] Specifically, generating the panoramic image specifically includes:

[0057] Initialize the panoramic image I p = I1, and construct the set of images to be stitched Set R = {I2, I3, ……, I N};

[0058] Stitch the images in the set of images to be stitched with the stitched panoramic image according to the grid position, and update the set of images to be stitched and the panoramic image at the same time;

[0059] In each stitching, construct the straight line matching set corner point matching and the texture matching set of the images to be stitched m and p, and construct the matching point set Realize image stitching through image registration;

[0060] Straight line matching set:

[0061] Among them, the straight line feature set of the image m is N m is the number of straight lines, and the nth straight line is: ρ m,n , θ m,n are respectively the perpendicular distance from the straight line n to the origin and the angle with the horizontal axis, are respectively the starting point and the ending point of the projection of the straight line n along a certain direction;

[0062] Before each stitching, contrast screening is introduced to prevent the dynamic flame area from affecting the accuracy of feature point extraction. Specifically, the contrast map is calculated for the images m and p to be stitched, and thresholding is used for screening to eliminate the low-contrast areas of the images to be stitched.

[0063] The image matching includes the identity matrix H p,m Calculate: And perform coordinate transformation ), where is the pixel value of the projected image I m of, is the inverse matrix of H p,m .

[0064] After generating the panoramic image, smoke removal processing is required for the panoramic image. The specific process of the smoke removal processing includes:

[0065] Traverse each pixel position (x, y) in the panoramic image, construct a set of fire images Set(x, y) corresponding to the pixel position, and the fire images in the set of fire images all cover the pixel position (x, y) in the panoramic image;

[0066] Derive the smoke mask and the clear mask

[0067]

[0068] where is the smoke mask of the t-th fire image, recording the smoke position information in the t-th fire image;

[0069] Traverse each pixel position (x, y) in the panoramic image;

[0070] When that is, at least one fire image has no smoke at this position, then select a pixel from the smokeless image for replacement:

[0071] I p (x, y) = I` t (x, y), where t ∈ Set(x, y) and at the same time,

[0072] If that is, there are multiple fire images that have no smoke at this position, then pixel fusion is used to further reduce noise;

[0073] When then global smoke processing is performed to determine the dominant smoke type And determine the smoke removal algorithm based on the dominant smoke type; if white smoke is dominant, perform brightness smoothing on the smoke area, if black smoke is dominant, perform brightness enhancement on the smoke area, and if yellow smoke is dominant, perform median filtering on the smoke area;

[0074]

[0075] wherein, is the smoke type at the (x, y) position of t fire scene images, and Majority is the voting function.

[0076] In the process of smoke removal of the present invention, a two-stage smoke removal method is adopted, which can effectively improve the authenticity and clarity of the panoramic image, while taking into account the processing efficiency and the robustness of complex scenes. The core idea of the first stage is to give priority to using the local image without smoke for pixel replacement. The biggest advantage of this method is that it directly retains the real information in the fire scene image, does not need to rely on complex smoke removal algorithms, and avoids damaging the details of the original image. By checking all local images covering a certain position of the panoramic image, if there is at least one local image without smoke at this position, the pixel value without smoke can be directly used to replace the pixel in the smoke area, ensuring the clarity and authenticity of this area. At the same time, when multiple local images are all without smoke at this position, further through pixel fusion or selecting the pixel with the highest quality image, the noise can be significantly reduced and the visual effect of the image can be improved. The present invention makes full use of the redundancy of multi-view data to ensure the high-quality reconstruction of the panoramic image in the area without local smoke.

[0077] Specifically, judging whether there is smoke at a certain position of the panoramic image depends on the comprehensive information of the smoke mask of the local image. If all local images covering this position detect smoke, it is reasonably inferred that the smoke at this position is a global or inevitable smoke phenomenon; if there is at least one local image without smoke, it means that the smoke is a local phenomenon and can be replaced by the image without smoke. The processing of the first stage is more efficient, which can significantly reduce the amount of calculation while retaining the details of the real scene.

[0078] For those areas where all local images have smoke, through the unified smoke removal algorithm in the second stage, targeted processing methods can be adopted for different types of smoke, so as to achieve the global smoke removal effect. To sum up, the smoke removal algorithm of the present invention can not only give priority to using real images to optimize the panoramic image, but also make up for the lack of local data through algorithms when necessary.

[0079] Based on Collect a set of smoke sequence images: wherein, is the s-th frame image of the q-th smoke area, N frIndicates the number of frames collected;

[0080] Among them, the specific acquisition process of the smoke sequence image set includes: calibrating the smoke center point and the circumscribed rectangle in the panoramic image to obtain the smoke area set: Set area ={(x q ,y q ,x q,max ,y q,max ,x q,min ,y q,min )|q = 1, 2,......, N q}; where, (x q ,y q ) is the center point of the smoke area, x q,max ,y q,max ,x q,min ,y q,min are the horizontal and vertical coordinates of the maximum and minimum values of the circumscribed rectangle, and N q is the number of smoke areas;

[0081] The ground control system plans the flight path for the UAV according to the center point of the smoke area, sorts according to the area size of the smoke area, and preferentially captures the smoke areas with larger areas; the UAV arrives at the center point of the smoke area according to the planned flight path to collect the corresponding smoke sequence images:

[0082] Use the optical flow method to track the starting point of the smoke based on the smoke sequence image set, and construct the first weight map of the panoramic image based on the Gaussian distribution based on all the starting points of the smoke;

[0083] Fuse the panoramic image and the first weight image to obtain a region-enhanced panoramic image, and perform multi-target recognition on the region-enhanced panoramic image;

[0084] The target recognition network includes a backbone network, a multi-scale fusion module, a feature enhancement module, and a detection head. The backbone network obtains 4 multi-scale feature maps F1 - F4; the multi-scale fusion module includes an edge feature module, an edge enhancement module, and an edge fusion module

[0085] The edge feature module is defined as:

[0086] F marginal = Concat(Conv 1*1 (F1), Upsampling(Conv 1*1 (F1)));

[0087]

[0088] Among them, F marginal and Indicates the edge feature map and the masked edge feature map. Conv represents the convolution operation, and the subscript is the convolution kernel size;

[0089] The multi-scale fusion module is defined as:

[0090]

[0091] where \(i\in[2,4]\), is the feature map of the \(i\)-th fusion module, and represent element-wise multiplication and addition respectively;

[0092]

[0093]

[0094] where, is the feature map of the 1st fusion module;

[0095] The feature enhancement module is defined as:

[0096]

[0097] where, are the two inputs of the enhancement module, \(F\) en is the output of the enhancement module, are respectively the four sub-feature maps obtained by dividing the shallow feature map by channels, are respectively the intermediate feature maps of the feature enhancement module;

[0098] The feature enhancement module has three groups of feature maps. The first group of inputs: The second group of inputs: is the output feature map obtained by the feature enhancement module processing the first group of inputs. The third group of inputs: is the output feature map obtained by the feature enhancement module processing the second group of inputs;

[0099] The three groups of output feature maps of the feature enhancement module are input to the detection head to obtain the object recognition result of the panoramic image;

[0100] The object recognition result includes human targets, open fire targets, and plant targets. When there is no open fire target in the smoke starting point area, mark this area as a potential fire source target.

[0101] Constructing the risk level map based on the panoramic image specifically includes:

[0102] Construct an initial risk level matrix based on the target recognition results; calculate the mutual influence factors based on the spatial relationship between the targets, update the initial risk level matrix, and obtain the risk level map;

[0103] Each value in the initial risk level matrix is the risk value corresponding to the target covered by the pixel position;

[0104] The mutual influence factors include the diffusion factor between the open fire target and the plant target, the potential combustion factor between the potential fire source target and the plant target, and the safety distance factor between the personnel target and the open fire target. The above mutual influence factors use the Gaussian kernel function to map the relationship between the distance between the targets and the danger value.

[0105] Use the risk level map to control the drone to execute multiple tasks

[0106] The GCS serves as the control center and is responsible for task allocation of the drone according to the risk level map, the real-time state of the drone (and the task priority. The GCS prioritizes the allocation of drones to execute fire extinguishing and personnel evacuation tasks according to the location of the high-risk area in the risk level map, and preferentially schedules the drone closest to the target task to reduce the task response time. When the risk level in a certain area increases, the GCS schedules the drones in the surrounding areas to cooperate and complete the emergency tasks first.

[0107] The present invention also provides an embedded drone on-board multi-task control system, which includes:

[0108] Image acquisition module: The image acquisition module uses the drone to acquire multiple fire scene images and detect smoke information;

[0109] Panoramic image generation module: The panoramic image generation module generates a panoramic image based on the multiple fire scene images;

[0110] Risk level map generation module: The risk level map generation module constructs a risk level map based on the panoramic image;

[0111] Task control module: The task control module uses the risk level map to control the drone to execute multiple tasks.

[0112] The present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and operable on the processor. When the processor executes the computer program, it implements the above-mentioned embedded drone on-board multi-task control method.

[0113] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned embedded drone on-board multi-task control method.

[0114] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above various methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0115] As described above, the above are only the specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An embedded UAV airborne multi-task control method, characterized in that, The control method includes the following steps: S1: Use a drone to collect multiple fire scene images and detect smoke information; S2: Generate a panoramic image based on the multiple fire scene images; After generating the panoramic image, it is necessary to perform de-smoking processing on the panoramic image. During the de-smoking process, a two-stage removal method is adopted. Traverse each pixel position (x, y) in the panoramic image, construct a set of fire scene images Set(x, y) corresponding to the pixel position, and deduce the smoke mask and clear mask of the panoramic image. and clear mask Among them, is the smoke mask of the t-th fire scene image, recording the smoke position information in the t-th fire scene image; Traverse each pixel position (x, y) in the panoramic image. When select a pixel from the smokeless image for replacement. At the same time, if perform pixel fusion; When Global smoke processing is performed to determine the dominant smoke type, and a smoke removal algorithm is determined based on the dominant smoke type; S3: Construct a risk level map based on the panoramic image; Based on The collected smoke sequence image set includes: the drone reaches the center point of the smoke area according to the planned flight path to collect the corresponding smoke sequence images: is the s-th frame image of the q-th smoke area, N fr represents the number of collected frames. The starting point of the smoke is tracked by the optical flow method according to the smoke sequence image set, and the first weight map of the panoramic image based on the Gaussian distribution is constructed based on all the starting points of the smoke; the panoramic image and the first weight image are fused to obtain a region-enhanced panoramic image, and multi-object recognition is performed on the region-enhanced panoramic image; an initial risk level matrix is constructed based on the target recognition result; the mutual influence factor is calculated based on the spatial relationship between the targets, and the initial risk level matrix is updated to obtain a risk level map; S4: Use the risk level map to control the drone to perform multiple tasks; Detecting smoke information includes: performing color space conversion on the fire scene image; extracting the fire smoke position and type based on color features, contrast features, and texture sparsity features; The texture sparsity feature is a texture feature extracted by local binary pattern, and the texture sparsity is defined as: where LBP is a local texture feature operator, Image(i, j) is the image to be processed, Ω represents the number of pixels in the window, and Ω(x, y) represents the pixel set in the window; The white smoke detection conditions are as follows: Among them, are the minimum and maximum thresholds of the white smoke brightness, and S(x, y) is the saturation value of the pixel point in the HSV color space. are the white smoke saturation threshold, contrast threshold, and sparsity threshold, respectively; The black smoke detection conditions are as follows: Among them, is the maximum black smoke brightness, are the black smoke saturation threshold, contrast threshold, and sparsity threshold respectively; The detection conditions for yellow smoke are as follows: Among them, are the minimum and maximum thresholds of the yellow smoke hue, are the saturation threshold and the sparsity threshold of the yellow smoke respectively.

2. The embedded UAV airborne multi-task control method according to claim 1, characterized in that, Generating a panoramic image specifically includes: initializing the panoramic image I p = I1, constructing an image set Set R = {I2, I3,......, I N}; Stitch the images in the set of images to be stitched with the stitched panoramic image according to the grid position, and update the set of images to be stitched and the panoramic image at the same time; In each splicing, construct a set of line matches for the images m and p to be spliced Corner point matching and a set of texture matches and construct a set of matching points Through image registration, image splicing is achieved; Set of line matches: Among them, the set of linear features of image m is N m which is the number of straight lines, and the nth straight line is: ρ m,n ,θ m,n which are respectively the perpendicular distance from the straight line n to the origin and the angle with the horizontal axis, which are respectively the starting point and the ending point of the projection of the straight line n along a certain direction.

3. The method for multi-task control based on an embedded airborne drone according to claim 2, wherein Calibrate the smoke center point and the circumscribed rectangle in the panoramic image to obtain a set of smoke areas: Set area = {(x q , y q , x q,max , y q,max , x q,min , y q,min ) | q = 1, 2,......, N q}; where, (x q , y q ) is the center point of the smoke area, x q,max , y q,max , x q,min , y q,min are the maximum and minimum horizontal and vertical coordinates of the circumscribed rectangle, and N q is the number of smoke areas.

4. An embedded UAV airborne multi-task control system for implementing an embedded UAV airborne multi-task control method according to any one of claims 1 to 3, characterized in that, The system includes: Image acquisition module: The image acquisition module uses a drone to collect multiple fire scene images and detect smoke information; Panoramic image generation module: The panoramic image generation module generates a panoramic image based on the multiple fire scene images; Risk level map generation module: The risk level map generation module constructs a risk level map based on the panoramic image; Task control module: The task control module uses the risk level map to control the drone to perform multiple tasks.

5. An electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements an embedded drone airborne multi-task control method according to any one of claims 1-3.

6. A computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, it implements an embedded drone airborne multi-task control method according to any one of claims 1-3.

Images