Unmanned aerial vehicle forest fire smoke detection method based on deep learning method

Abstract

The invention relates to the technical field of image recognition, in particular to an unmanned aerial vehicle forest fire smoke detection method based on a deep learning method. The method comprises the following steps: acquiring a scene image of a forest fire; constructing a flame smoke recognition network model, and inputting the scene image into the flame smoke recognition network model for training to obtain a first model; storing the first model in the unmanned aerial vehicle; if flames or smog are detected, the coordinate position of the unmanned aerial vehicle and the coordinate position of the smog or flame in a pixel coordinate system are obtained and marked as a first coordinate and a second coordinate respectively, and the position of the flame or the smog is calculated and predicted according to the first coordinate and the second coordinate to serve as a third coordinate of the position of the flame or the smog; and sending the third coordinate and an alarm signal to the background. According to the invention, conversion is carried out through different coordinates, so that the specific positions of the flame and the smoke can be rapidly predicted, the rapid reporting of a fire point is realized, and the disaster relief efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method for detecting forest fire smoke using an unmanned aerial vehicle (UAV) based on a deep learning method. Background Art

[0002] Nowadays, fire smoke recognition technology based on computer vision is steadily advancing in forest fire prevention and forest fire fighting activities. Video surveillance equipment such as drones, law enforcement instruments, and surveillance cameras have been gradually applied in the intelligent pilot projects of forest police and forest management systems, providing a relevant foundation for the realization of drone forest fire smoke recognition systems based on deep learning. Although intelligent fire monitoring technology based on computer vision has been developed, in the current application practice of drone forest fire smoke detection based on deep learning, there are still the following problems to be solved, mainly including: (1) The need for manual review of background videos. At present, video surveillance of fires is responsible for assisting alarms and issuing warnings. The collected video images need to rely on manual review. The workload of supervisors is large, repetitive, and due to the influence of human factors, they may miss the video, and the review efficiency is low.

[0003] (2) The control effect is highly subjective and has poor real-time performance. In the initial stage of a fire, the open flame is not obvious, and most cases require manual intervention. The search for fire disasters is still in the offline mode, especially when the fire first occurs. It is difficult to determine the origin of the fire disaster. Due to human subjectivity and complexity, many operations cannot be indexed and modularized. The effect depends entirely on the personal experience and subjectivity of the backend staff, and false alarms and missed alarms often occur. At present, some neural network-based systems have realized forest fire detection in the cloud, but most detection methods use video surveillance and other methods to identify flames. During the initial stage of a fire, the flame is usually weak, and the characteristics of the smoke are more obvious than the flame characteristics. This has led to the existing forest fire detection system being more inclined to detect large fires in dense jungles, which makes it difficult to extinguish fires, and often leads to fires being taken seriously only when they become serious.

[0004] (3) During the fire identification process, only flames and thick smoke are detected, while the white smoke in the initial stage of the fire is ignored. In actual detection, white smoke is often identified as normal clouds, which leads to insufficient judgment of the situation in the early stage of the fire. Summary of the invention

[0005] In view of the above-mentioned defects, the purpose of the present invention is to propose a UAV forest fire smoke detection method based on a deep learning method, so as to reduce the amount of manual participation and improve the intelligence of forest fire detection.

[0006] To achieve this purpose, the present invention adopts the following technical solution: a method for detecting forest fire smoke using a drone based on a deep learning method, comprising the following steps:

[0007] Get forest fire scene images;

[0008] A flame and smoke recognition network model is constructed, and scene images are input into the flame and smoke recognition network model for training to obtain a first model;

[0009] storing the first model in the drone;

[0010] A drone is driven above the forest to obtain a forest image, and the forest image is input into a first model of the drone for identification. If flame or smoke is detected, the coordinate position of the drone and the coordinate position of the smoke or flame in the pixel coordinate system are obtained and marked as the first coordinate and the second coordinate respectively. The position of the flame or smoke is predicted based on the first coordinate and the second coordinate, and the third coordinate is used as the position of the flame or smoke. The third coordinate and an alarm signal are sent to the background.

[0011] Preferably, before executing the flame and smoke recognition network model, it is also necessary to perform data enhancement operation on the scene image.

[0012] The data enhancement operations are as follows:

[0013] Construct four marking matrices, which are used to mark the positions of the four scene images that need to be cropped and retained, respectively, where the retained positions are set to 1 and the cropped positions are set to 0;

[0014] Assign a label value to each scene image;

[0015] Create a new image, fill pixels in the new image and fuse label values ​​according to the label matrix and the four scene images to generate a new scene image.

[0016] Preferably, the pixel filling formula is as follows:

[0017] P=M a *x a +M b *x b +M c *x c +M d *x d ;

[0018] Among them, M a 、M b 、M c 、M dThey are the labeling matrix of the first scene image, the labeling matrix of the second scene image, the labeling matrix of the third scene image, the labeling matrix of the fourth scene image, and x a 、x b 、x c 、x d are respectively the image data of the first scene image, the image data of the second scene image, the image data of the third scene image, the image data of the fourth scene image, and * represents the pixel multiplication of the matrix;

[0019] The formula for label value fusion is as follows:

[0020] Q=λ a *y a +λ b *y b +λ c *y c +λ d *y d ;

[0021] λ a , b , c , d Respectively represent the weight value of the first scene image, the weight value of the second scene image, the weight value of the third scene image, and the weight value of the fourth scene image, y a ,y b ,y c ,y d They are the label value of the first scene image, the label value of the second scene image, the label value of the third scene image, and the label value of the fourth scene image respectively.

[0022] Preferably, the flame and smoke recognition network model is composed of a backbone network, a neck neural network and a prediction head network connected in sequence;

[0023] The recognition process of the flame and smoke recognition network model is as follows:

[0024] The scene image is input into the backbone network to extract the target multi-scale features of the scene image and obtain a feature image;

[0025] In the neural network of the neck of the feature image input value, the pyramid pooling method is used to upsample and fuse different feature images;

[0026] The prediction head network predicts the result of the fused feature image.

[0027] Preferably, the backbone network comprises a convolutional layer, an MP layer and a residual ELAN module connected in sequence;

[0028] The number of convolutional layers is 3, and each convolutional layer is provided with a 3x3 convolution kernel;

[0029] The MP layer performs the following operations: performing maximum pooling on the scene image processed by the convolution layer and then passing it through a convolution kernel activation function to extract the first feature;

[0030] The scene image processed by the convolution layer is subjected to image feature extraction using the convolution kernel plus activation function method to achieve the extraction of the second feature;

[0031] Performing weighted fusion on the first feature and the second feature to obtain a fused feature image;

[0032] The residual ELAN module performs the following operations:

[0033] Four CBS modules are used to extract fine-grained features from feature images;

[0034] A single CBS module is used to extract coarse-grained features from feature images;

[0035] Perform weighted fusion of fine-grained features and coarse-grained features;

[0036] Global average pooling is used to set the feature size of the feature image to 1x1;

[0037] Use 1D convolution to calculate the channel weights of the feature image, and then use the Sigmoid function to implement weight mapping to obtain the processing result;

[0038] The processed result is multiplied with the original scene image to obtain the final feature image.

[0039] Preferably, the steps of storing the first model in the drone are as follows:

[0040] According to the network structure of the first model, define the corresponding network model structure in the ONNX model;

[0041] According to the calculation logic of the first model and the nodes with parameters, a directed graph structure is formed;

[0042] Modify the information of each node in the directed graph structure so that the size information of each node corresponds to the number of layers in the first model;

[0043] Transfer the weight information from the first model training process to the ONNX model;

[0044] The ONNX model is deployed on the FPGA embedded device of the drone.

[0045] Preferably, the step of calculating and predicting the position of the flame or smoke according to the first coordinate is as follows:

[0046] Get the world coordinates (X, Y, Z) of the drone and set the camera optical center coordinate system (X c ,Y c ,Z c ) and the world coordinate system (X, Y, Z) of the drone are 1, and the first transformation relationship between the camera optical center coordinate system and the world coordinate system of the drone is obtained, where the first transformation relationship is as follows:

[0047]

[0048] R is a 3*3 matrix indicating that the camera can change in three different dimensions, T is a 3*1 matrix indicating the distance change between the camera coordinate system and the drone coordinate system at a fixed distance, and t is the average height of the trees in the forest that the drone passes through;

[0049] According to the imaging relationship between the camera and the image, the image plane coordinate system (x, y) and the camera optical center coordinate system (X c ,Y c ,Z c ), wherein the second transformation relationship is as follows:

[0050]

[0051] Among them, the value of f is Zt, and Z is the altitude coordinate of the drone;

[0052] The captured image pixel coordinate system (u, v) and the image plane coordinate system (x, y) are transformed to obtain a third transformation relationship, wherein the third transformation relationship is as follows:

[0053]

[0054] Combining the first transformation relationship, the second transformation relationship and the third transformation relationship, we get the fourth transformation relationship:

[0055]

[0056] Substitute the first coordinate into (X, Y, Z) of the fourth transformation relationship, and substitute the second coordinate into (u, v) of the fourth transformation relationship to obtain the third coordinate (x, y).

[0057] One of the above technical solutions has the following advantages or beneficial effects: In the present invention, the model is trained for recognition through the scene image containing the forest fire to obtain the first model, and the first model is stored on the drone. When the drone is inspected, the forest can be photographed by the camera, and the photographed image is directly input into the first model, and the first model is used to detect in real time whether there is a fire or smoke. Automatic identification of fire is achieved without manual inspection, which improves the intelligence of detection. At the same time, since there is a position difference between the image taken by the camera and the drone, the present invention also converts through different coordinates to quickly predict the specific location of the flame and smoke, thereby realizing the rapid reporting of the fire point and improving the efficiency of disaster relief. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a flow chart of an embodiment of the method of the present invention.

[0059] Figure 2 It is a structural schematic diagram of a flame and smoke recognition network model in an embodiment of the method of the present invention. DETAILED DESCRIPTION

[0060] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0061] In the description of the embodiments of the present invention, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0062] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, the meaning of "plurality" is two or more. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0063] like Figures 1-2 As shown, a method for detecting forest fire smoke using a drone based on a deep learning method comprises the following steps:

[0064] Get forest fire scene images;

[0065] First, the drone route is planned. The drone's flight route needs to pass through the central axis of the forest and fly at a fixed time. Secondly, the camera collects video image information to construct a forest fire smoke scene image with fire smoke conditions, and the flames and smoke in the scene image dataset are calibrated using anchor frames.

[0066] A flame and smoke recognition network model is constructed, and scene images are input into the flame and smoke recognition network model for training to obtain a first model;

[0067] storing the first model in the drone;

[0068] A drone is driven above the forest to obtain a forest image, and the forest image is input into a first model of the drone for identification. If flame or smoke is detected, the coordinate position of the drone and the coordinate position of the smoke or flame in the pixel coordinate system are obtained and marked as the first coordinate and the second coordinate respectively. The position of the flame or smoke is predicted based on the first coordinate and the second coordinate, and the third coordinate is used as the position of the flame or smoke. The third coordinate and an alarm signal are sent to the background.

[0069] In the present invention, the model is trained for recognition through the scene image containing the forest fire to obtain the first model, and the first model is stored on the drone. When the drone is inspected, the forest can be photographed through the camera, and the photographed image is directly input into the first model, and the first model is used to detect in real time whether there is a fire or smoke. Automatic identification of fire is achieved without manual inspection, which improves the intelligence of detection. At the same time, since there is a position difference between the image taken by the camera and the drone, the present invention also converts through different coordinates to quickly predict the specific location of the flame and smoke, thereby realizing the rapid reporting of the fire point and improving the efficiency of disaster relief.

[0070] Preferably, before executing the flame and smoke recognition network model, it is also necessary to perform data enhancement operation on the scene image.

[0071] The data enhancement operations are as follows:

[0072] Construct four marking matrices, which are used to mark the positions of the four scene images that need to be cropped and retained, respectively, where the retained positions are set to 1 and the cropped positions are set to 0;

[0073] Assign a label value to each scene image;

[0074] Create a new image, fill pixels in the new image and fuse label values ​​according to the label matrix and the four scene images to generate a new scene image.

[0075] Through the above steps, the four images can be effectively stitched together to form a new scene image containing more semantic information. Finally, the size of the scene image is set to 600x600 to obtain a fixed-size RGB matrix image. This method can increase the diversity of data, enhance the robustness of the model, and strengthen the effect of the batch normalization layer.

[0076] Preferably, the pixel filling formula is as follows:

[0077] P=M a *x a +M b *x b +M c *x c +M d *x d ;

[0078] Among them, M a 、M b 、M c 、M d They are the labeling matrix of the first scene image, the labeling matrix of the second scene image, the labeling matrix of the third scene image, the labeling matrix of the fourth scene image, and x a 、x b 、x c 、x d are respectively the image data of the first scene image, the image data of the second scene image, the image data of the third scene image, the image data of the fourth scene image, and * represents the pixel multiplication of the matrix;

[0079] The formula for label value fusion is as follows:

[0080] Q=λ a *y a +λ b *y b +λ c *y c +λ d *y d ;

[0081] λ a , b , c , d Respectively represent the weight value of the first scene image, the weight value of the second scene image, the weight value of the third scene image, and the weight value of the fourth scene image, y a ,y b,y c ,y d They are the label value of the first scene image, the label value of the second scene image, the label value of the third scene image, and the label value of the fourth scene image respectively.

[0082] like Figure 2 As shown, the flame and smoke recognition network model is composed of a backbone network, a neck neural network and a prediction head network connected in sequence;

[0083] The recognition process of the flame and smoke recognition network model is as follows:

[0084] The scene image is input into the backbone network to extract the target multi-scale features of the scene image and obtain a feature image;

[0085] Through multi-scale feature extraction, features of flames or smoke at various scales can be obtained. These rich features can describe the target more comprehensively, thereby improving the accuracy of target recognition.

[0086] In the neural network of the neck of the feature image input value, the pyramid pooling method is used to upsample and fuse different feature images; in this step, the CBS module uses a convolution with a kernel size of 4 and a step length of 4 to upsample feature images of different depths, fuse the output feature images of each layer, and output the fused feature images to the prediction head network. The purpose of using large kernel convolution in this method is to increase the receptive field of the neck network, so that the neck network module is conducive to fully integrating the features of different dimensions extracted from the backbone network, thereby improving the final detection success rate.

[0087] The prediction head network predicts the result of the fused feature image.

[0088] Different feature image dimensions will be passed to different detection head modules, and finally the outputs of detection heads of different sizes will be integrated into the same image to achieve the final image smoke and flame detection.

[0089] Preferably, the backbone network comprises a convolutional layer, an MP layer and a residual ELAN module connected in sequence;

[0090] The number of convolutional layers is 3, and each convolutional layer is provided with a 3x3 convolution kernel;

[0091] The MP layer performs the following operations: performing maximum pooling on the scene image processed by the convolution layer and then passing it through a convolution kernel activation function to extract the first feature;

[0092] The scene image processed by the convolution layer is subjected to image feature extraction using the convolution kernel plus activation function method to achieve the extraction of the second feature;

[0093] Performing weighted fusion on the first feature and the second feature to obtain a fused feature image;

[0094] In the present invention, the construction of the MP layer is improved by referring to the residual network. The feature image is processed by two branches. The first branch performs maximum pooling on the feature map and then extracts effective features through the convolution kernel activation function. The second branch further extracts image features by using the convolution kernel plus activation function method on the feature map, and finally obtains the weighted fused feature image by using the image weighted summation method. The image enters the residual ELAN module proposed by the present invention.

[0095] The present invention can extract image features from different angles and scales by combining the method of pooling operation and convolution plus activation function. The first feature (extracted by pooling operation) mainly focuses on the maximum value in the local area, emphasizing texture and edge information; while the second feature (extracted only by convolution plus activation function) retains more space and detail information. By weighted fusion of the first feature and the second feature, a fused feature that combines the advantages of the two features can be obtained. This fused feature not only contains the local detail information of the image, but also emphasizes the global structure and texture information of the image, which helps to improve the accuracy and robustness of the model.

[0096] The residual ELAN module performs the following operations:

[0097] Four CBS modules are used to extract fine-grained features from feature images;

[0098] A single CBS module is used to extract coarse-grained features from feature images;

[0099] Perform weighted fusion of fine-grained features and coarse-grained features;

[0100] Global average pooling is used to set the feature size of the feature image to 1x1;

[0101] Use 1D convolution to calculate the channel weights of the feature image, and then use the Sigmoid function to implement weight mapping to obtain the processing result;

[0102] The processed result is multiplied with the original scene image to obtain the final feature image.

[0103] The designed residual ELAN module is designed for the forest images taken by drones, which are characterized by many small target categories and unclear edge features of the identified objects. The method first uses a dual-branch structure to process the image, in which the first branch uses 4 CBS modules to realize the feature extraction of fine-grained images, and the other branch uses a single CBS module to realize the feature extraction of coarse-grained features of feature images. The two realize the model's processing and superposition capabilities of feature images of different sizes through weighted summation, effectively improving the problem of feature aliasing caused by small drone aerial images and too many target objects. The feature image processed by the above steps uses global average pooling to set the feature size of the feature map to 1x1 to achieve global context fusion, and then uses adaptive convolution size and 1D convolution to calculate the channel weight of the feature image, and then uses Sigmoid function to realize weight mapping. The processing result is multiplied with the original scene image to improve the weight of the feature of the effective feature channel in the scene image.

[0104] The final multiplication operation is the processing of the reference residual network structure, which multiplies the particle features with the weighted information of the effective feature channel and the original input scene image, and finally obtains the final feature image with effective feature channel information weights. This method can effectively enhance the edge features of the object to be identified, which is helpful to solve the problem of smoke being difficult to detect in the image.

[0105] Preferably, the steps of storing the first model in the drone are as follows:

[0106] According to the network structure of the first model, define the corresponding network model structure in the ONNX model;

[0107] According to the calculation logic of the first model and the nodes with parameters, a directed graph structure is formed;

[0108] Modify the information of each node in the directed graph structure so that the size information of each node corresponds to the number of layers in the first model;

[0109] Transfer the weight information from the first model training process to the ONNX model;

[0110] The ONNX model is deployed on the FPGA embedded device of the drone.

[0111] Since the first model is equipped with a residual ELAN module and a MAP layer, it requires a lot of computing power when in use. Due to its size, the drone cannot be equipped with more processors, so the first model cannot be directly installed on the drone. The first model needs to be packaged using the ONNX method before it can be placed in the drone's artificial intelligence computing chip. By packaging the first model and deploying it to the drone's artificial intelligence computing chip using the ONNX method, cross-framework compatibility, cross-platform portability, optimization and acceleration, broad community support, lightweight and high efficiency, as well as scalability and flexibility can be achieved, making it possible for the first model to run efficiently on edge devices such as drones.

[0112] Preferably, the step of calculating and predicting the position of the flame or smoke according to the first coordinate is as follows:

[0113] The optical center coordinate system refers to the camera coordinate system constructed with the optical center, i.e., the focal center of the pinhole camera model, as the origin and the optical axis of the camera as Xc, Yc, and Zc. In the present invention, since the camera can be fixed in the drone, the camera coordinate system and the world coordinate system of the drone can be set to 1.

[0114] Get the world coordinates (X, Y, Z) of the drone and set the camera optical center coordinate system (X c ,Y c ,Z c ) and the world coordinate system (X, Y, Z) of the drone are 1, and the first transformation relationship between the camera optical center coordinate system and the world coordinate system of the drone is obtained, where the first transformation relationship is as follows:

[0115]

[0116] R is a 3*3 matrix indicating that the camera can change in three different dimensions, T is a 3*1 matrix indicating the distance change between the camera coordinate system and the drone coordinate system at a fixed distance, and t is the average height of the trees in the forest that the drone passes through;

[0117] According to the imaging relationship between the camera and the image, the image plane coordinate system (x, y) and the camera optical center coordinate system (X c ,Y c ,Z c ), wherein the second transformation relationship is as follows:

[0118]

[0119] Among them, the value of f is Zt, and Z is the altitude coordinate of the drone;

[0120] The captured image pixel coordinate system (u, v) and the image plane coordinate system (x, y) are transformed to obtain a third transformation relationship, wherein the third transformation relationship is as follows:

[0121]

[0122] Combining the first transformation relationship, the second transformation relationship and the third transformation relationship, we get the fourth transformation relationship:

[0123]

[0124] Substitute the first coordinate into (X, Y, Z) of the fourth transformation relationship, and substitute the second coordinate into (u, v) of the fourth transformation relationship to obtain the third coordinate (x, y).

[0125] After completing the coordinate system calibration of the detected forest area during the first UAV flight, the complete UAV coordinate system and image pixel coordinate system are obtained. When the UAV detects a fire during its patrol flight, the coordinate system can be converted to obtain the position information (second coordinate) of the fire anchor frame in the pixel coordinate system. The coordinate information of the fire occurrence location (third coordinate) can be obtained based on the pixel coordinate system constructed during the first flight.

[0126] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0127] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A UAV forest fire smoke detection method based on deep learning method, characterized in that: The steps include: Get forest fire scene images; A flame and smoke recognition network model is constructed, and scene images are input into the flame and smoke recognition network model for training to obtain a first model; storing the first model in the drone; A drone is driven above the forest to obtain a forest image, and the forest image is input into a first model of the drone for identification. If flame or smoke is detected, the coordinate position of the drone and the coordinate position of the smoke or flame in the pixel coordinate system are obtained and marked as the first coordinate and the second coordinate respectively. The position of the flame or smoke is predicted based on the first coordinate and the second coordinate, and the third coordinate is used as the position of the flame or smoke. The third coordinate and an alarm signal are sent to the background.

2. The method for detecting forest fire smoke using a drone based on a deep learning method according to claim 1, characterized in that: Before executing the flame and smoke recognition network model, it is necessary to perform data enhancement on the scene image. The data enhancement operations are as follows: Construct four marking matrices, which are used to mark the positions of the four scene images that need to be cropped and retained, respectively, where the retained positions are set to 1 and the cropped positions are set to 0; Assign a label value to each scene image; Create a new image, fill pixels in the new image and fuse label values ​​according to the label matrix and the four scene images to generate a new scene image.

3. The method for detecting forest fire smoke using a drone based on a deep learning method according to claim 2, characterized in that: The pixel filling formula is as follows: P=M a *x a +M b *x b +M c *x c +M d *x d ; Among them, M a 、M b 、M c 、M d They are the labeling matrix of the first scene image, the labeling matrix of the second scene image, the labeling matrix of the third scene image, the labeling matrix of the fourth scene image, and x a 、x b 、x c 、x d are respectively the image data of the first scene image, the image data of the second scene image, the image data of the third scene image, the image data of the fourth scene image, and * represents the pixel multiplication of the matrix; The formula for label value fusion is as follows: Q=λ a *y a +λ b *y b +λ c *y c +λ d *y d ; λ a , b , c , d Respectively represent the weight value of the first scene image, the weight value of the second scene image, the weight value of the third scene image, and the weight value of the fourth scene image, y a ,y b ,y c ,y d They are the label value of the first scene image, the label value of the second scene image, the label value of the third scene image, and the label value of the fourth scene image respectively.

4. The method for detecting forest fire smoke using a drone based on a deep learning method according to claim 1, characterized in that: The flame and smoke recognition network model is composed of a backbone network, a neck neural network and a prediction head network connected in sequence; The recognition process of the flame and smoke recognition network model is as follows: Input the scene image into the backbone network, extract the target multi-scale features of the scene image, and obtain the feature image; In the neural network of the neck of the feature image input value, the pyramid pooling method is used to upsample and fuse different feature images; The prediction head network predicts the result of the fused feature image.

5. The method for detecting forest fire smoke using a drone based on a deep learning method according to claim 4, characterized in that: The backbone network includes a convolutional layer, an MP layer and a residual ELAN module connected in sequence; The number of convolutional layers is 3, and each convolutional layer is provided with a 3x3 convolution kernel; The MP layer performs the following operations: performing maximum pooling on the scene image processed by the convolution layer and then passing it through a convolution kernel activation function to extract the first feature; The scene image processed by the convolution layer is subjected to image feature extraction using the convolution kernel plus activation function method to achieve the extraction of the second feature; Performing weighted fusion on the first feature and the second feature to obtain a fused feature image; The residual ELAN module performs the following operations: Four CBS modules are used to extract fine-grained features from feature images; A single CBS module is used to extract coarse-grained features from feature images; Perform weighted fusion of fine-grained features and coarse-grained features; Global average pooling is used to set the feature size of the feature image to 1x1; Use 1D convolution to calculate the channel weights of the feature image, and then use the Sigmoid function to implement weight mapping to obtain the processing result; The processed result is multiplied with the original scene image to obtain the final feature image.

6. The method for detecting forest fire smoke using a drone based on a deep learning method according to claim 5, characterized in that: The steps to store the first model in the drone are as follows: According to the network structure of the first model, define the corresponding network model structure in the ONNX model; According to the calculation logic of the first model and the nodes with parameters, a directed graph structure is formed; Modify the information of each node in the directed graph structure so that the size information of each node corresponds to the number of layers in the first model; Transfer the weight information from the first model training process to the ONNX model; The ONNX model is deployed on the FPGA embedded device of the drone.

7. The method for detecting forest fire smoke using a drone based on a deep learning method according to claim 1, characterized in that: The steps of calculating and predicting the position of the flame or smoke according to the first coordinate are as follows: Get the world coordinates (X, Y, Z) of the drone and set the camera optical center coordinate system (X c ,Y c ,Z c ) and the world coordinate system (X, Y, Z) of the drone are 1, and the first transformation relationship between the camera optical center coordinate system and the world coordinate system of the drone is obtained, where the first transformation relationship is as follows: R is a 3*3 matrix indicating that the camera can change in three different dimensions, T is a 3*1 matrix indicating the distance change between the camera coordinate system and the drone coordinate system at a fixed distance, and t is the average height of the trees in the forest that the drone passes through; According to the imaging relationship between the camera and the image, the image plane coordinate system (x, y) and the camera optical center coordinate system (X c ,Y c ,Z c ), wherein the second transformation relationship is as follows: Among them, the value of f is Zt, and Z is the altitude coordinate of the drone; The captured image pixel coordinate system (u, v) and the image plane coordinate system (x, y) are transformed to obtain a third transformation relationship, wherein the third transformation relationship is as follows: Combining the first transformation relationship, the second transformation relationship and the third transformation relationship, we get the fourth transformation relationship: Substitute the first coordinate into (X, Y, Z) of the fourth transformation relationship, and substitute the second coordinate into (u, v) of the fourth transformation relationship to obtain the third coordinate (x, y).

Images