Identification and processing method of intelligent unmanned aerial vehicle fire behavior identification device

Abstract

The invention discloses a recognition and processing method of an intelligent unmanned aerial vehicle fire behavior recognition device. The system belongs to the field of image processing and computer vision and comprises an image acquisition system and an image processing system. The image acquisition system is provided with a plurality of high-definition cameras; meanwhile, an integrated thermal imaging sensor collects thermal radiation data, and thermal images are generated through preprocessing; the system adopts a multi-frame image fusion and super-resolution reconstruction technology, and the definition and details of the image are improved. According to the method, advanced features in the image are further extracted by combining self-supervision, unsupervised learning and an adversarial network model, and richer feature representation is provided for fire identification; according to the invention, the pixels of unmanned aerial vehicle fire source identification are upgraded, the accuracy and reliability of fire source detection are improved, and the method is more suitable for complex and changeable environments; in addition, due to the fact that the system has good robustness, the system can work stably under different environment conditions, and the system is beneficial to quickly responding to fire behaviors, reducing loss and improving rescue efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and computer vision, and relates to an intelligently driven unmanned aerial vehicle fire identification device; specifically, it relates to an identification and processing method of an intelligent unmanned aerial vehicle fire identification device. Background Art

[0002] In recent years, fires have occurred frequently, especially in complex environments such as forests, mountains and cities, and fire monitoring and emergency response have become major challenges. Although existing drone image recognition technology has made significant progress, the precision and accuracy of drone recognition may still be affected in complex environments such as lighting changes, high temperatures, and high smoke, which may lead to longer drone identification time or misidentification. Summary of the invention

[0003] In view of the above problems, the purpose of the present invention is to propose an identification and processing method for an intelligent unmanned aerial vehicle fire identification device; through the technology of combining multi-frame image fusion and ultra-high resolution reconstruction, improvements are made in pixels, the clarity and details of the image are improved, and then through the technology of combining self-supervised learning, unsupervised learning and adversarial network (GAN), high-level features are extracted and a comprehensive and detailed feature representation is formed, thereby improving the generalization ability and robustness of the model, significantly improving the accuracy and reliability of fire source detection, and being more suitable for complex and changeable environments such as fire monitoring and urban fire fighting; in addition, thanks to the good robustness of the system, it can work stably under different environmental conditions, which is helpful to quickly respond to fires, reduce losses and improve rescue efficiency.

[0004] The technical solution of the present invention is: an identification and processing method of an intelligent unmanned aerial vehicle fire identification device according to the present invention, and its operation steps are as follows:

[0005] (1): Collect real-time images and perform data preprocessing;

[0006] (2): Combine multi-frame image fusion with super-resolution reconstruction to further improve image clarity;

[0007] (3): Feature extraction is performed through self-supervised learning and unsupervised learning to learn the intrinsic structure and feature representation of the image; a generative adversarial network (GAN) model is trained to generate new features, and the features obtained from self-supervised learning are combined with the image features generated by GAN to obtain enhanced feature representation, thereby forming a comprehensive and detailed feature representation;

[0008] (4): The fused features are input into the trained classification model for training. The drone uses the trained features to identify the fire source and output information such as the location and range of the fire source.

[0009] Furthermore, in step (1), the real-time image is captured by an integrated high-definition camera and a thermal imaging sensor.

[0010] Furthermore, in step (1), the implementation process of the preprocessing is as follows:

[0011] The original image is de-noised to reduce random errors and interference in the image and improve image quality. For sparse or missing data points, the Lagrange interpolation algorithm (see the appendix for related codes) is used to estimate the missing values, and then the useful information in the image is enhanced through contrast adjustment, histogram equalization and other methods.

[0012] Furthermore, in step (2), the multi-frame image fusion-super-resolution reconstruction technology is implemented as follows:

[0013] (2.1) The image processing system fuses the collected visible light image and thermal imaging image, uses the temperature information provided by thermal imaging to supplement the details in the visible light image, and generates a low-resolution image of the current frame;

[0014] (2.2), given an input frame sequence image, is the low-resolution image of the current frame, are two adjacent low-resolution images of the current frame respectively; the input image is registered in the forward branch and the backward branch. Taking the forward branch as an example, the specific expression is:

[0015]

[0016] In the formula, After the optical flow estimation, the feature map is It is the feature map output by the optical flow fusion module. is the feature map after the auxiliary mechanism, S(·), W(·), T(·), C(·), and R(·) represent the optical flow estimation module, optical flow fusion module, auxiliary mechanism, convolution operation, and feature correction module, respectively. It represents the characteristic graph after the forward distribution is standardized at time t, and it is the characteristic graph after adding the auxiliary mechanism;

[0017] The algorithm flow of the backward branch is the same as that of the forward branch. represents the feature map after backward distribution calibration at time t, They serve as the backward hidden state and forward hidden state of the support frame at the next moment, and also serve as the basis of the fusion module;

[0018] (2.3) The fusion module uses the early fusion method, which is expressed as:

[0019]

[0020] In the formula, F fusion is the fusion result, [·] represents concatenation, and the input is the low-resolution image of the current frame and the feature map after alignment of each branch;

[0021] The upsampling part uses sub-pixel convolution, which is expressed as:

[0022]

[0023] In the formula, U(·) represents the upsampling module, which includes multiple convolutional feature extractions and a sub-pixel convolution upsampling. The input is the result of the fusion module. is the output high-resolution image of the current frame;

[0024] (2.4) The image feature encoding layer uses a multi-layer RRDB (Residual-in-Residual Dense Block) module stacking method to extract features. The RRDB module combines the dense connection mechanism with the residual connection strategy with learnable weighted coefficients.

[0025] Specifically, each RRDB module contains three Dense Blocks with the same structure. Each DenseBlock consists of a 3×3 convolution layer, a Leaky ReLU activation function, and dense connections. The output feature map of the Dense Block is superimposed on the input feature map through a learnable weighting factor, which helps to effectively alleviate the gradient vanishing problem that may occur in the dense connection architecture, thereby improving the overall expressiveness and stability of the network.

[0026] (2.5) Then, the feature map size is doubled by two bilinear upsampling (or corresponding interpolation techniques), and the high-resolution image is finally improved by combining the encoding operation, thereby improving the content fidelity of the image;

[0027] (2.6) The discriminator adopted in the present invention is based on the relative discriminator network architecture proposed in the Real-ESRGAN method. This architecture uses the concept of relative discrimination to convert the original binary super-resolution effect evaluation for the entire image into a binary reconstruction effect evaluation for each pixel in the image. At the same time, a U-Net network structure with jump connections is adopted, which can ultimately output the discrimination result of the reconstruction effect for each pixel in the image. In addition, spectral normalization is introduced as a network regularization term to further improve the discrimination ability of the discriminator.

[0028] Furthermore, the implementation process in step (3) is as follows:

[0029] (3.1) Consider two images with shared content. Ia and Ib ; By using the SuperPoint algorithm, an encoder focused on general fine-grained matching, the feature point sets of the two images are extracted, denoted as a={a1…,a n} and b={b1,…,b m};

[0030] Where n and m are I a and I b The number of identified feature points; each feature point is associated with a SuperPoint local descriptor z∈R C At the same time, the image processing system further integrates a pre-defined set of fire feature points, which are used as part of the training benchmark data to optimize the accuracy of feature matching in fire detection tasks; by analyzing the quantity and quality of feature points and comparing them with known fire feature patterns, more accurate fire status determination can be achieved;

[0031] (3.2) The position of the feature point is encoded by position embedding, and the MLP layer (a fully connected neural network) is used to further refine the position feature; the normalized position feature of the feature point is represented as h∈R C ;

[0032] (3.3), for I a and I b DINOv2 feature map extraction is performed on the two images;

[0033] Among them, the DINOv2 feature map is an image-based model that encodes rough but extensive visual knowledge, which can provide a rough signal for capturing the general similarity between SuperPoint feature points; the DINOv2 feature map is interpolated using the position of the SuperPoint feature point to obtain the DINOv2 descriptor of each feature point, expressed as g∈R C '; For the i-th feature point in set a, the three features are represented by z i a ,h i a , g i a , correspondingly represents the features of the feature points in set b; the model aims to estimate the correspondence between two sets of feature points, so as to extract features;

[0034] (3.4) Construct four feature point association graphs: two inter-image graphs and two intra-image graphs;

[0035] Among them, the image-to-image graph represents Ia and I b The connectivity between the feature points, that is, G a→b and G b→a , the internal graph of the image represents the connectivity between feature points belonging to the same image, that is, G a and G b ;

[0036] (3.5) In the process of information propagation, the constructed graph contains multiple modules, each of which is equipped with two layers of attention mechanisms: self-attention and cross-attention, to refine the feature representation of feature points; for the cross-attention mechanism, the feature points ai Information is only collected from the potential matching set pruned by DINOv2 in set b. In addition, a new position-guided attention mechanism is introduced, which processes the position feature h and the local descriptor z of the feature point separately. In this module, the position information acts as a spatial context and is not included in the final local descriptor representation. Based on these new elements, in the attention layer, the feature point a is used as the local descriptor. i Take as an example, and give the attention update formula:

[0037]

[0038] Among them, the attention has a residual connection, which integrates the attention update value The symbol ← is the update operation, MLP is a fully connected neural network symbol, and [·|·] is the channel cascade;

[0039] Attention update value The calculation of is as follows:

[0040]

[0041] In the formula, the feature point a i The feature similarity between the feature points connected to the source in the graph is expressed as S containing K feature points, i.e., k S , is the attention query, k S is the key of attention, v S is the attention value, c is the dimension of the feature vector, R C is a space with C real dimensions, and softmax is the activation function;

[0042] Attention Query Key k S Sum value v S The calculation formula is as follows:

[0043]

[0044] k S =Ek (z S +h S )+Y k ∈R K×C

[0045] v S =E v (z S )+Y v ∈R K×C

[0046] In the formula, RC is a space with C real dimensions, R K×C For key k S The dimension, E q , E k , E v are their respective weight matrices, Y q , Y k , Y v are the corresponding bias vectors, is the local descriptor of the i-th feature point in set a, is the position feature of the i-th feature point in set a, z S is the local descriptor of all feature points, h S is the position feature of all feature points;

[0047] In the above three equations, the query and key are calculated by fusing the local descriptor and the position feature, and the value is only converted from the local descriptor; the weights (E) and biases (Y) are used to map features to the query, key, and value in the attention, and are not shared between different attention layers; in the self-attention G a and G b In the cross attention G a→b and G b→a In , S contains the feature points recognized by DINO;

[0048] (3.6), according to the two images I a and I b , and their respective feature point sets, use the optimized feature point representation to construct the pairwise similarity matrix S∈R n×m , S i,j Defined as feature point a i and b i The similarity measure between them is calculated as in, and Represent the descriptors of the i-th and j-th feature points in sets a and b respectively; then, the Sinkhorn algorithm (see the appendix for relevant code) is used to further refine the similarity measure to obtain the matching matrix m∈[0,1]n×m , where m i,j Indicates feature point a i and b j The matching probability between ;

[0049] (3.7) The matched feature point descriptors are fused into a global feature representation F and input into the pre-trained GAN model: G theta (F), where G theta is a generator network with parameters θ;

[0050] (3.8) Generate new image features using adversarial networks

[0051] (3.9) The real image feature F and the generated image feature Fusion, get the enhanced feature representation F':

[0052] (3.10) Determine whether the obtained enhanced features are within the range of the training benchmark answer database. If not, return to step 2). If yes, go to step 11);

[0053] (3.11), output enhanced features.

[0054] Furthermore, in step (4), the implementation process is as follows:

[0055] The enhanced features are input into the trained classification model for training; the classification model learns how to map the input features to the corresponding fire source category and location information; the model parameters are iteratively optimized through a large amount of training data until convergence, and then used to identify and locate the fire source; finally, the drone outputs information such as the location and range of the fire source, which can be further used to guide rescue operations.

[0056] The beneficial effects of the present invention are: 1. Through the dual enhancement of multi-frame image fusion and ultra-high resolution reconstruction technology, the drone can capture fire source images from multiple angles and time periods for identification, thereby effectively reducing the information loss or misjudgment caused by a single perspective or a single shot, making the frame, shape and other detailed features of the fire source more obvious, thereby improving the accuracy and reliability of fire source identification; at the same time, the feature correction of the backbone network is performed through an auxiliary mechanism to solve the calculation error caused by the long-term dependency in the deep network architecture and the information bias problem caused by the recycling of features; this method can significantly reduce the frequency of texture synthesis artifacts that appear in the process of reconstructing the image, thereby improving the realism and detail restoration of the reconstructed image; this The scheme not only strengthens the model's ability to manage feature flows, but also improves its robustness and accuracy in processing complex image tasks; 2. The present invention adopts a combination of self-supervised learning and generative adversarial networks to generate additional training images, effectively expanding the diversity and richness of the database; these high-quality training images cover more possible fire source forms and scene changes, thereby providing the model's generalization ability and robustness, which helps to improve the accuracy and stability of fire source identification; 3. Thanks to its strong environmental adaptability and robustness, drones perform well in practical application scenarios such as fire monitoring and urban firefighting. They can operate stably under different environmental conditions, accurately capture fire source information, respond quickly to fire conditions, and reduce losses. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is an operation flow chart of the present invention;

[0058] Figure 2 is a flow chart of a multi-frame image fusion-super-resolution reconstruction technology in an embodiment of the present invention;

[0059] Figure 3 is a diagram of the image registration network structure in an embodiment of the present invention;

[0060] Figure 4 It is a definition diagram of the attention layer in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The specific technical scheme of the present invention is further described in detail below with reference to specific examples.

[0062] As shown in the figure, the identification and processing method of an intelligent unmanned aerial vehicle fire identification device described in the present invention is shown in the figure; the unmanned aerial vehicle includes an image acquisition system and an image processing system; the image acquisition system is equipped with multiple high-definition cameras, which can capture images in an all-round and seamless manner; at the same time, the integrated thermal imaging sensor collects thermal radiation data and generates thermal images after preprocessing.

[0063] The image processing system fuses the collected visible light image and thermal imaging image, and uses the temperature information provided by thermal imaging to supplement the details in the visible light image. At the same time, through multi-frame image fusion-super-resolution reconstruction technology, image registration is performed in the forward branch and the backward branch, and feature extraction is performed by stacking multiple layers of RRDB modules to further improve the clarity and details of the image.

[0064] The specific implementation process of the multi-frame image fusion-super-resolution reconstruction technology is as follows:

[0065] (1) The image processing system fuses the collected visible light image and thermal imaging image, uses the temperature information provided by the thermal imaging to supplement the details in the visible light image, and generates a low-resolution image of the current frame;

[0066] (2) Given an input frame sequence image, is the low-resolution image of the current frame, They are two adjacent low-resolution images of the current frame;

[0067] The input image is registered in the forward branch and the backward branch. Taking the forward branch as an example, the specific expression is:

[0068]

[0069] In the formula, Feature map after optical flow estimation; It is the feature map output by the optical flow fusion module; is the feature map after the auxiliary mechanism, S(·), W(·), T(·), C(·), and R(·) represent the optical flow estimation module, optical flow fusion module, auxiliary mechanism, convolution operation, and feature correction module respectively; It represents the characteristic graph after the forward distribution is standardized at time t, and it is the characteristic graph after adding the auxiliary mechanism;

[0070] The algorithm flow of the backward branch is the same as that of the forward branch. represents the feature map after backward distribution calibration at time t, They serve as the backward hidden state and forward hidden state of the support frame at the next moment, and also serve as the basis of the fusion module;

[0071] (3) The fusion module uses the early fusion method, which is expressed as:

[0072]

[0073] In the formula, F fusionis the fusion result, [·] represents concatenation, and the input is the low-resolution image of the current frame and the feature map after calibration of each branch; the upsampling part uses the sub-pixel convolution method, and its expression is:

[0074]

[0075] Where U(·) represents the upsampling module, which includes multiple convolutional feature extractions and a sub-pixel convolutional upsampling, and the input is the result of the fusion module. is the output high-resolution image of the current frame;

[0076] (4) The image feature encoding layer uses a multi-layer RRDB (Residual-in-Residual Dense Block) module stacking method to extract features.

[0077] Among them, the RRDB module combines the dense connection mechanism with the residual connection strategy with learnable weight coefficients;

[0078] Specifically, each RRDB module contains three Dense Blocks with the same structure. Each DenseBlock consists of a 3×3 convolution layer, a Leaky ReLU activation function, and dense connections. The output feature map of the Dense Block is superimposed on the input feature map through a learnable weighting factor, which effectively alleviates the gradient vanishing problem that may occur in the dense connection architecture, thereby improving the overall expressiveness and stability of the network.

[0079] 5) Then, the feature map size is doubled by two bilinear upsampling (or corresponding interpolation techniques), and combined with the encoding operation, the high-resolution image is finally improved, thereby improving the content fidelity of the image.

[0080] 6) The discriminator used in the present invention is based on the relative discriminator network architecture proposed in the Real-ESRGAN method. The architecture uses the concept of relative discrimination to convert the original binary super-resolution effect evaluation for the entire image into a binary reconstruction effect evaluation for each pixel in the image; at the same time, a U-Net network structure incorporating jump connections is adopted, which can ultimately output a discrimination result for the reconstruction effect of each pixel in the image; in addition, the present invention introduces spectral normalization as a network regularization term to further improve the discrimination ability of the discriminator;

[0081] The CPU uses self-supervised learning and unsupervised learning technology to learn the intrinsic structure and feature representation of the image, further extract high-level features in the image, and provide richer feature representation for fire identification. In addition, the CPU uses a pre-trained generative adversarial network (GAN) model to generate new features to enhance the diversity and richness of the data set, and then fuses the real and generated features to obtain enhanced feature representation, forming a comprehensive and detailed feature representation, improving the generalization ability and robustness of the model for more accurate fire condition determination. The specific implementation steps are as follows:

[0082] 1) Consider two images I with shared content a and I b ; By using the SuperPoint algorithm, an encoder focused on general fine-grained matching, the feature point sets of the two images are extracted, denoted as a={a1…,a n} and b={b1,…,b m}, where n and m are I a and I b The number of identified feature points; each feature point is associated with a SuperPoint local descriptor z∈R C At the same time, the image processing system further integrates a pre-defined set of fire feature points, which are used as part of the training benchmark data to optimize the accuracy of feature matching in fire detection tasks; by analyzing the quantity and quality of feature points and comparing them with known fire feature patterns, more accurate fire status determination can be achieved;

[0083] 2) Encode the position of the feature point through position embedding, and use the MLP layer (a fully connected neural network) to further refine the position feature; the normalized position feature of the feature point is represented as h∈R C ;

[0084] 3) Further, a and I b DINOv2 feature map extraction is performed on the two images; the DINOv2 feature map is an image-based model that encodes rough but extensive visual knowledge, providing a rough signal for capturing the general similarity between SuperPoint feature points; the DINOv2 feature map is interpolated using the position of the SuperPoint feature point to obtain the DINOv2 descriptor of each feature point, denoted as g∈R C' ; For the i-th feature point in set a, the three features are represented as z i a ,h i a , g ia , correspondingly represents the features of the feature points in set b; the model aims to estimate the correspondence between two sets of feature points, so as to extract features;

[0085] 4) Construct four feature point association graphs: two inter-image graphs and two intra-image graphs; among them, the inter-image graph represents I a and I b The connectivity between the feature points, that is, G a→b and G b→a , the internal graph of the image represents the connectivity between feature points belonging to the same image, that is, G a and G b ;

[0086] 5) In the process of information propagation, the constructed graph contains multiple modules, each of which is equipped with two layers of attention mechanisms: self-attention and cross-attention, to refine the feature representation of feature points; for the cross-attention mechanism, feature point a i Information is only collected from the potential matching set pruned by DINOv2 in set b. In addition, a new position-guided attention mechanism is introduced, which processes the position feature h and the local descriptor z of the feature point separately. In this module, the position information acts as a spatial context and is not included in the final local descriptor representation. Based on these new elements, the feature point a is used as the local descriptor in the attention layer. i Take as an example, and give the attention update formula:

[0087]

[0088] Among them, the attention has a residual connection, which integrates the attention update value The symbol ← is the update operation, MLP is a fully connected neural network symbol, and [·|·] is the channel cascade;

[0089] Attention update value The calculation of is as follows:

[0090]

[0091] Among them, the feature point a i The feature similarity between the feature points connected to the source in the graph is expressed as S containing K feature points, i.e., k S , is the attention query, k S is the key of attention, v S is the attention value, c is the dimension of the feature vector, R C is a space with C real dimensions, and softmax is the activation function;

[0092] Attention Query Key kS Sum value v S The calculation formula is as follows:

[0093]

[0094] k S =E k (z S +h S )+Y k ∈R K×C

[0095] v S =E v (z S )+Y v ∈R K×C

[0096] Among them, R C is a space with C real dimensions, R K×C For key k S The dimension, E q , E k , E v are their respective weight matrices, Y q , Y k , Y v are the corresponding bias vectors, is the local descriptor of the i-th feature point in set a, is the position feature of the i-th feature point in set a, z S is the local descriptor of all feature points, h S is the position feature of all feature points;

[0097] In the above three equations, the query and key are calculated by fusing the local descriptor and the position feature, and the value is only converted from the local descriptor; the weights (E) and biases (Y) are used to map features to the query, key, and value in the attention, and are not shared between different attention layers; in the self-attention G a and G b In the cross attention G a→b and G b→a In , S contains the feature points recognized by DINO;

[0098] 6) According to the two images I a and I b , and their respective feature point sets, use the optimized feature point representation to construct the pairwise similarity matrix S∈R n×m , S i,j Defined as feature point a i and b i The similarity measure between them is calculated as in, and Denote the descriptors of the i-th and j-th feature points in sets a and b, respectively. Next, the Sinkhorn algorithm (see the Appendix for the relevant code) is used to further refine the similarity measure, thereby obtaining the matching matrix m∈[0,1] n×m , where m i,j Indicates feature point a i and b j The matching probability between ;

[0099] 7) The matched feature point descriptors are fused into a global feature representation F and input into the pre-trained GAN model: G theta (F), where G theta is a generator network with parameters θ;

[0100] 8) Generate new image features using adversarial networks

[0101] 9) Combine the real image feature F with the generated image feature Fusion, get enhanced feature representation F:

[0102] 10) Determine whether the obtained enhanced features are within the range of the training benchmark answer database. If not, return to step 2). If yes, go to step 11);

[0103] 11) Output enhanced features.

[0104] Finally, the fused features are input into the trained classification model to identify and locate the fire source, so that the drone can output information such as the location and range of the fire source.

Claims

1. An identification and processing method for an intelligent unmanned aerial vehicle fire identification device, characterized in that: The operation steps are as follows: Step (1): collecting real-time images and performing data preprocessing; Step (2): combining multi-frame image fusion with super-resolution reconstruction technology to improve image clarity; Step (3): Feature extraction through self-supervised learning-unsupervised learning to learn the intrinsic structure and feature representation of the image; Train a generative adversarial network model to generate new features, combine the features obtained from self-supervised learning with the image features generated by the adversarial network model to obtain enhanced feature representation, and form a comprehensive feature representation; Step (4): The fused features are input into the trained classification model for training. The drone uses the trained features to identify the fire source and output information on the location and range of the fire source.

2. The identification and processing method of the intelligent unmanned aerial vehicle fire identification device according to claim 1 is characterized in that: In step (1), the real-time image is captured by an integrated high-definition camera and a thermal imaging sensor.

3. The identification and processing method of the intelligent unmanned aerial vehicle fire identification device according to claim 1 is characterized in that: In step (1), the implementation process of the data preprocessing is as follows: removing noise from the original image to reduce random errors and interference in the image; For sparse or missing data points, the Lagrange interpolation algorithm is used to estimate the missing values, and then the useful information in the image is enhanced through contrast adjustment and histogram equalization.

4. The identification and processing method of the intelligent unmanned aerial vehicle fire identification device according to claim 1 is characterized in that: In step (2), the process of combining multi-frame image fusion with super-resolution reconstruction technology is as follows: (2.1) The image processing system fuses the collected visible light image and thermal imaging image, uses the temperature information provided by thermal imaging to supplement the details in the visible light image, and generates a low-resolution image of the current frame; (2.2), given an input frame sequence image, is the low-resolution image of the current frame, They are two adjacent low-resolution images of the current frame; (2.3) The fusion module uses the early fusion method, which is expressed as: In the formula, F fusion is the fusion result; [·] represents concatenation, the input is the low-resolution image of the current frame and the feature map after calibration of each branch; the upsampling part uses sub-pixel convolution, and its expression is: In the formula, U(·) represents the upsampling module, which includes multiple convolutional feature extractions and a sub-pixel convolution upsampling. The input is the result of the fusion module. is the output high-resolution image of the current frame; (2.4) The image feature encoding layer uses a multi-layer RRDB module stacking method to extract features. The RRDB module combines the dense connection mechanism with the residual connection strategy with learnable weight coefficients. (2.5) The feature map size is doubled by two bilinear upsamplings, and encoding operations are performed based on the extracted features to improve the high-resolution image; (2.6) The discriminator adopts the relative discriminator network architecture of Real-ESRGAN, which transforms the original binary super-resolution effect evaluation for the entire image into a binary reconstruction effect evaluation for each pixel in the image. At the same time, the discriminator combines the U-Net network structure to output the discrimination result of the reconstruction effect for each pixel in the image.

5. The identification and processing method of the intelligent unmanned aerial vehicle fire identification device according to claim 1 is characterized in that: In step (2.2), the input image is registered in the forward branch and the backward branch. Taking the forward branch as an example, its expression is: In the formula, After the optical flow estimation, the feature map is It is the feature map output by the optical flow fusion module. is the feature map after the auxiliary mechanism, S(·), W(·), T(·), C(·), and R(·) represent the optical flow estimation module, optical flow fusion module, auxiliary mechanism, convolution operation, and feature correction module, respectively. It represents the characteristic graph after the forward distribution is standardized at time t, and it is the characteristic graph after adding the auxiliary mechanism; The algorithm flow of the backward branch is the same as that of the forward branch. represents the feature map after backward distribution calibration at time t, They serve as the backward hidden state and forward hidden state of the support frame at the next moment, and also serve as the basis of the fusion module.

6. The identification and processing method of the intelligent unmanned aerial vehicle fire identification device according to claim 1 is characterized in that: In step (2.4), specifically, each RRDB module contains three Dense Blocks with the same structure. Each Dense Block consists of a 3×3 convolutional layer, a Leaky ReLU activation function, and dense connections. The output feature map of the Dense Block is superimposed on the input feature map through a learnable weighting factor.

7. The identification and processing method of the intelligent unmanned aerial vehicle fire identification device according to claim 1 is characterized in that: In step (3), the feature extraction process is as follows: (3.1) Consider two images I with shared content a and I b ; By using the SuperPoint algorithm to extract the feature point sets of the two images, they are represented as a={a1…,a n } and b={b1,…,b m }, where n and m are I a and I b The number of identified feature points, where each feature point is associated with a SuperPoint local descriptor z∈R C associated; At the same time, the image processing system integrates a pre-defined set of fire feature points, analyzes the quantity and quality of the feature points, and compares them with known fire feature patterns to detect the accuracy of feature matching in fire tasks and determine the fire status; (3.2) The position of the feature point is encoded by position embedding, and the position feature is refined using the MLP layer. The normalized position feature of the feature point is represented as h∈R C ; (3.3), for I a and I b DINOv2 feature map is extracted from two images; wherein, DINOv2 feature map is an image-based model for capturing the general similarity between SuperPoint feature points; By interpolating the DINOv2 feature map, we obtain the DINOv2 descriptor of each SuperPoint feature point, represented as g∈R C' ; For the i-th feature point in set a, the three features are represented as z i a ,h i a , g i a , correspondingly represents the features of the feature points in set b; (3.4) Construct four feature point association graphs: two inter-image graphs and two intra-image graphs; the inter-image graph represents I a and I b The connectivity between the feature points, that is, G a→b and G b→a , the inner graph of the image represents the connectivity between feature points belonging to the same image, that is, G a and G b ; (3.5) In the process of information propagation, the constructed graph contains multiple modules with self-attention and cross-attention to refine the feature representation of feature points; for the cross-attention mechanism, the feature points ai Only collect information from the potential matching set pruned by DINOv2 in set b; where feature point a i Take as an example, and give the attention update formula: In the formula, the attention has a residual connection, which integrates the attention update value The symbol ← is the update operation, MLP is a fully connected neural network symbol, and [·|·] is the channel cascade; Attention update value Δzia The calculation of is as follows: In the formula, the feature point a i The feature similarity between the feature points connected to the source in the graph is expressed as S containing K feature points, i.e., k S , is the attention query, k S is the key of attention, v S is the attention value, c is the dimension of the feature vector, R C is a space with C real dimensions, and softmax is the activation function; (3.6), according to the two images I a and I b , and their respective feature point sets, use the optimized feature point representation to construct the pairwise similarity matrix S∈R n×m , used to measure the similarity between two feature points, the calculation formula is: In the formula, S i,j Defined as feature point a i and b i The similarity measure between and Represent the descriptors of the i-th and j-th feature points in sets a and b respectively; Next, the Sinkhorn algorithm is used to refine the similarity measure and obtain the matching matrix m∈[0,1] n×m , Among them, m i,j Indicates feature point a i and b j The matching probability between ; (3.7) The matched feature point descriptors are fused into a global feature representation F and input into the pre-trained adversarial network model: G theta (F), where G theta is a generator network with parameters θ; (3.8) Generate new image features using adversarial networks (3.9) The real image feature F and the generated image feature Fusion, to obtain enhanced feature representation F′ (3.10) Determine whether the obtained enhanced features are within the range of the training benchmark answer database. If not, return to step 2). If yes, go to step 11); (3.11), output enhanced features.

8. The identification and processing method of the intelligent unmanned aerial vehicle fire identification device according to claim 1 is characterized in that: In step (3.3), in the 3 equations, the query and key are calculated by fusing the local descriptor and the position feature, and the value is only converted from the local descriptor; the weights (E) and biases (Y) are used to map features to queries, keys, and values ​​in the attention, and are not shared between different attention layers; In the self-attention G a and G b In the cross attention G a→b and G b→a In , S contains the feature points recognized by DINO.

9. The identification and processing method of the intelligent unmanned aerial vehicle fire identification device according to claim 1 is characterized in that: In step (3.5), the attention query Key k S Sum value v S The calculation formula is as follows: k S =E k (z S +h S )+Y k ∈R K×C v S =Yes v (z S )+Y v ∈R K×C In the formula, R C is a space with C real dimensions, R K×C For key k S The dimension, E q , E k , E v are their respective weight matrices, Y q , Y k , Y v are the corresponding bias vectors, is the local descriptor of the i-th feature point in set a, is the position feature of the i-th feature point in set a, z S is the local descriptor of all feature points, h S is the position feature of all feature points.

10. The identification and processing method of the intelligent unmanned aerial vehicle fire identification device according to claim 1 is characterized in that: The implementation process of step (4) is as follows: the enhanced features are input into a trained classification model for training; the classification model learns to map the input features to the corresponding fire source category and location information; The model parameters are iteratively optimized through training data until convergence, and then used to identify and locate the fire source; finally, the drone outputs information on the location and range of the fire source, which can be used to guide rescue operations.

Images