Outdoor fireworks intelligent identification method and system for high-altitude inspection of unmanned aerial vehicle

Abstract

The present disclosure is suitable for the field of unmanned aerial vehicle fireworks identification, and provides an outdoor fireworks intelligent identification method and system for high-altitude inspection of unmanned aerial vehicles. Based on the optical flow field, the present application extracts a dynamic physical index group including a region diffusion index, a flow field vortex index, a motion direction index, a flicker entropy, and a texture blur degree. The essential motion characteristics of fireworks, which are different from interference sources such as clouds, fog, and reflections, are described from multiple dimensions including smoke diffusion, rotation, direction consistency, open fire flicker, and texture blur degree. In addition, a joint gating collaborative filtering method is used to fuse the original single-frame detection confidence and the above dynamic indexes to calculate a joint score, thereby early screening non-firework targets. Furthermore, the video frame sequence, the deep feature, and the physical index group are respectively encoded in a high-dimensional space, and a multi-modal spatio-temporal fusion method is used for global temporal fusion, so that the system can make a judgment based on the complete spatio-temporal evolution process of the fireworks rather than isolated frame information.

Description

Technical Field

[0001] This application belongs to the field of drone smoke and fire recognition, and specifically relates to an intelligent method for outdoor smoke and fire recognition for drone high-altitude inspection. Background Technology

[0002] With the widespread application of drone technology in high-altitude inspection scenarios such as forest fire prevention and urban security, the need for rapid and accurate automatic identification of smoke and fire anomalies is becoming increasingly urgent. Currently, common smoke and fire detection methods mainly include traditional physical sensors (such as smoke and heat detectors) and video-based manual monitoring. Both of these methods suffer from significant shortcomings in open outdoor environments and wide field-of-view situations, such as response delays or limited field of view. In recent years, deep learning-based target detection algorithms (such as the YOLO series and Faster R-CNN) have been gradually introduced into smoke and fire identification tasks, enabling the preliminary location of suspected smoke and fire areas through single-frame image analysis. However, most existing solutions rely solely on the appearance features of single-frame images for classification, failing to fully utilize key spatiotemporal dynamic information such as "smoke diffusion" and "flickering flames" in video sequences. Furthermore, they lack effective mechanisms to distinguish between common interference factors on drone platforms, such as camera movement, lighting changes, clouds, fog, and water surface reflections.

[0003] Therefore, the main technical problem faced by the existing technology is that, since it only relies on single-frame images for appearance feature discrimination, it lacks the modeling and fusion utilization of the essential dynamic laws of the movement, diffusion, vortex entrainment, brightness flickering, etc. of the smoke and fire targets in continuous time series. This makes it difficult to effectively distinguish real smoke and fire from various non-smoke and fire anomalies (such as clouds, fog, water vapor reflection, etc.) in complex outdoor environments (such as distant small targets, cloud and fog interference, sudden changes in lighting, etc.). As a result, the identification system has defects of high false alarm rate and high false alarm rate, which cannot meet the engineering application requirements of high reliability and low false alarm rate for UAV high-altitude inspection. Summary of the Invention

[0004] To address the aforementioned issues, this application provides an intelligent outdoor smoke and fire identification method for high-altitude drone inspections. This method significantly suppresses false alarms caused by non-smoke and fire anomalies such as clouds, water vapor, and reflections in complex outdoor environments, while enhancing the detection capability of real smoke and fire at long distances, with small sizes, and under background interference.

[0005] Optionally, an intelligent method for identifying outdoor smoke and fire during high-altitude drone inspections includes: The video stream of the drone inspection is acquired, and the video stream is decoded and keyframes are sampled to obtain the video frame sequence to be analyzed. The video frame sequence is input into the first deep learning detection network to locate candidate fireworks areas and generate preliminary detection results including target location, category and original confidence. Based on the preliminary detection results and video frame sequence, the optical flow field corresponding to each candidate fireworks area is calculated, and a set of indicators for characterizing the dynamic physical properties of the target is extracted based on the optical flow field. A joint gating collaborative filtering method is adopted, which integrates the original confidence level and the index group to calculate the joint score of each candidate fireworks region. Candidate fireworks regions with joint scores below a preset threshold are filtered as invalid targets, while candidate fireworks regions with joint scores above the preset threshold are retained as candidate targets. The video frame sequence containing the candidate target, the depth features extracted from the video frame sequence by the feature extraction network, and the index group are respectively encoded in high-dimensional space to obtain video frame representation, depth feature representation and physical index representation in a unified high-dimensional space. A multimodal spatiotemporal fusion method is adopted to globally fuse video frame representation, depth feature representation and physical index representation in time sequence to generate fused features. Based on the fused features, the final probability of the candidate target being real fireworks is calculated and output.

[0006] Furthermore, the set of indicators characterizing the dynamic physical properties of the target is calculated through the following steps: Based on the difference in optical flow components in the horizontal direction and the difference in optical flow components in the vertical direction of the pixels in the candidate region, the sum of the two means in the region is calculated to obtain the regional diffusion index used to characterize the diffusion trend of smoke. Based on the difference between the vertical optical flow component of the pixel in the candidate region and the horizontal optical flow component and the vertical optical flow component, the mean of the two differences is calculated to obtain the flow field vortex index used to characterize smoke vortices and / or entrainment motion. Based on the cosine value of the angle between the optical flow vector of each pixel in the candidate region and the optical flow vector of the previous frame, the mean value of the cosine value in the region is calculated to obtain the motion direction index used to characterize the continuous consistency of the target's motion direction. Based on the brightness change value of the same pixel position in the candidate region between adjacent frames, the histogram distribution of the brightness change value in the region is statistically analyzed and the information entropy is calculated to obtain the flicker entropy used to characterize the flickering of open flames and / or deflagration. Edge intensity is extracted from the image within the candidate region. Based on the average value of the extracted edge intensity, texture blur is obtained to characterize the difference between the texture of the semi-transparent smoke and the texture of the solid target.

[0007] Furthermore, the joint gating collaborative filtering method includes: Set dynamic gate judgment conditions. When the regional diffusion index, the flow field vortex index and the motion direction index all exceed their respective motion characteristic thresholds, the candidate pyrotechnic area is judged to pass through the dynamic gate. Set static gate judgment conditions. When the flicker entropy and texture ambiguity meet their respective appearance feature conditions, the candidate fireworks area is judged to pass through the static gate. Only when a candidate pyrotechnic region passes through both dynamic and static gates is its joint score calculated based on a weighted sum of the original confidence level, region diffusion index, motion direction index, and scintillation entropy, and compressed by an activation function; each weight coefficient is adaptively adjusted according to the degree of scene interference.

[0008] Furthermore, multimodal spatiotemporal fusion methods include: The video frame sequence containing the candidate target is input into the first coding network, which maps it to a high-dimensional space to generate a video frame sequence representation with location information. The vector sequence composed of the index group is input into the second encoding network and mapped to a high-dimensional space to generate a physical index representation with location information. The video frame sequence is input into a convolutional neural network to extract depth feature maps, and the depth feature map sequence is input into a third encoding network to generate a depth feature representation with location information. Video frame sequence representation, deep feature representation, and physical index representation are spliced ​​together in the same high-dimensional space and input into a fusion model based on a transform network to capture the self-dependencies within the sequence and the interdependencies between different representation sequences. The fused features output by the transformation network are passed sequentially through a fully connected layer and a flexible maximum activation function to obtain the final probability that the candidate target is a real firework.

[0009] Furthermore, after obtaining the video frame representation, depth feature representation, and physical index representation in a unified high-dimensional space, the result output is also included, which includes: Based on the final probability that the candidate target is a real firework and the preset alarm threshold, metadata including target bounding box, category, confidence level, timestamp and geographic coordinates is generated and reported or / and stored through application programming interface.

[0010] Furthermore, the regional diffusion index is calculated using the following method: The mean of the spatial difference values ​​of the horizontal optical flow component and the mean of the spatial difference values ​​of the vertical optical flow component within the candidate region are added together to obtain a value that characterizes the outward expansion and / or diffusion intensity of the target region.

[0011] Furthermore, the following method is used to calculate the vortex index of the flow field: The spatial difference between the vertical optical flow component and the horizontal optical flow component within the candidate region is subtracted from the spatial difference between the horizontal optical flow component and the vertical optical flow component. The resulting mean is used to characterize whether vortex flow and / or entrainment features exist in the target region.

[0012] Furthermore, when the first deep learning detection network outputs preliminary detection results, it records the position and size information of candidate fireworks regions with resolution and / or size lower than the preset small target threshold, as a key focus during optical flow field calculation.

[0013] 9. An intelligent outdoor smoke and fire detection system for high-altitude drone inspection, comprising: The video acquisition and preprocessing module is used to acquire the video stream of the UAV inspection, and to decode and sample key frames of the video stream to obtain the video frame sequence to be analyzed. The preliminary detection module is used to input the video frame sequence into the first deep learning detection network, locate the candidate fireworks area, and generate a preliminary detection result containing the location, category and original confidence of the area. The dynamic physical feature extraction module is used to calculate the optical flow field corresponding to each candidate fireworks area based on the preliminary detection results and video frame sequence, and extract a set of indicators to characterize the dynamic physical properties of the target based on the optical flow field. The joint gating collaborative filtering module is used to use the joint gating collaborative filtering method to fuse the original confidence and index group, calculate the joint score of each candidate fireworks region, filter out candidate fireworks regions with joint scores below a preset threshold as invalid targets, and retain candidate fireworks regions with joint scores above the preset threshold as candidate targets to be fused. The multi-source feature coding module is used to encode the video frame sequence containing the candidate target to be fused, the deep features extracted by the feature extraction network from the video frame sequence, and the index group in high-dimensional space to obtain video frame representation, deep feature representation and physical index representation in a unified high-dimensional space. The multimodal spatiotemporal fusion and determination module is used to globally fuse video frame representations, depth feature representations, and physical index representations in time sequence using a multimodal spatiotemporal fusion method, generate fused features, and calculate and output the final probability that the candidate target to be fused is a real firework based on the fused features.

[0014] A computer program product includes computer instructions that, when executed by a processor, specifically perform the steps of any of the methods described above.

[0015] Compared with the prior art, this application has the following advantages: This application extracts a set of dynamic physical indicators, including the region diffusion index, flow field vortex index, motion direction index, scintillation entropy, and texture blurring, from the optical flow field. It characterizes the essential motion features of smoke and fire from multiple dimensions, such as smoke diffusion, rotation, directional consistency, flame flickering, and texture blurring, distinguishing them from interference sources like clouds, fog, and reflections. Furthermore, it employs a joint gated collaborative filtering method, fusing the original single-frame detection confidence with the aforementioned dynamic indicators to calculate a joint score, thus filtering out non-smoke and fire targets at an early stage. Additionally, it performs high-dimensional spatial encoding on the video frame sequence, depth features, and physical indicator sets, and utilizes a multimodal spatiotemporal fusion method for global temporal fusion, enabling the system to make judgments based on the complete spatiotemporal evolution of smoke and fire rather than isolated frame information. The combined effect of the above-mentioned technical means enables this application to effectively capture and utilize the key dynamic patterns of smoke and fire in the time series, such as diffusion, entrainment, and flashing. This significantly suppresses false alarms of non-smoke and fire anomalies such as clouds, water vapor, and reflections in complex outdoor environments. At the same time, it enhances the ability to detect real smoke and fire at long distances, with small size, and under background interference, thus solving the problems of high false alarm rate and high false alarm rate in the prior art.

[0016] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of a method according to an embodiment of this application is shown; Figure 2 A flowchart of the multimodal spatiotemporal fusion algorithm analysis according to an embodiment of this application is shown; Figure 3 This illustrates a distant smoke recognition map from a drone's perspective, according to an embodiment of this application. Figure 4 An open flame diagram from a drone perspective according to an embodiment of this application is shown; Figure 5 A system framework diagram according to an embodiment of this application is shown. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] This application can be applied to high-altitude drone inspection scenarios such as forest fire prevention, urban safety monitoring, and power line inspection.

[0021] In the above scenario, drones equipped with cameras collect real-time video and need to automatically identify smoke or open flame anomalies in the footage. However, existing technologies typically use single-frame image target detection algorithms (such as the YOLO series), which rely solely on static appearance features for classification. This makes it difficult to utilize temporal dynamic information such as smoke diffusion and open flame flickering. Furthermore, it is prone to misreporting non-smoke / fire targets such as clouds, fog, and water surface reflections as smoke / fire, resulting in high false alarm and false negative rates. This fails to meet the requirements of low false alarm rate and high reliability in practical engineering.

[0022] To address the aforementioned technical problems, this application provides an intelligent outdoor smoke and fire detection method and system. Through combined gating collaborative filtering and multimodal spatiotemporal deep fusion, it effectively suppresses false alarms and missed alarms, improving detection accuracy and processing efficiency. The specific embodiments of this application are described in detail below with reference to the accompanying drawings.

[0023] The method in this embodiment includes the following steps S1 to S6, and the overall process is as follows: Figure 1 As shown, specifically: S1. Video stream access and preprocessing.

[0024] S1-1, Video stream data access.

[0025] This step aims to input the video stream data from the drone into a ground station or airborne edge computing device for processing. Conventional technical transmission methods can be used, supporting various video acquisition methods, including webcams (supporting Real-Time Streaming Protocol (RTSP) and Real-Time Messaging Protocol (RTMP)) or local video files (such as MP4 and AVI formats). The ground station or airborne edge computing device can automatically identify and process parameters such as resolution, frame rate, and compression format of the video stream, and decode them uniformly to ensure that subsequent analysis obtains standardized video frame data with correct format and qualified quality.

[0026] S1-2, Target recognition preprocessing.

[0027] This step aims to standardize the input video stream data. First, key parameters (resolution, frame rate, color space, bitstream format, etc.) can be detected. Then, frame extraction is performed based on actual business needs and detection algorithm efficiency. For example, 1 to 5 frames can be extracted from the original 25 frames per second video for analysis to improve processing speed and reduce meaningless redundant calculations.

[0028] After preprocessing, the video frame sequence to be analyzed is obtained.

[0029] S2. Initial target detection and motion feature analysis.

[0030] This step uses a deep learning detection network to locate candidate fireworks regions and extracts dynamic physical indicators such as region diffusion index, flow field vortex index, motion direction index, scintillation entropy, and texture ambiguity based on the optical flow field. The aim is to elevate static appearance discrimination to dynamic motion feature analysis, providing multi-dimensional evidence for distinguishing between fireworks and non-smoke. Specific sub-steps are as follows: S2-1, Target Detection and Preliminary Classification.

[0031] The video frame sequence to be analyzed obtained in S1 is input into the first deep learning detection network. The network automatically scans and locates all candidate smoke and fire areas that may be suspected of smoke, open flame or other suspicious objects in each frame. Preferably, the first deep learning detection network can use the YOLO series network, a real-time target detection network based on single forward propagation.

[0032] It should be noted that the candidate smoke and fire area refers to the image area that has a certain probability of being smoke or fire, as determined by the first deep learning detection network.

[0033] Each candidate fire area corresponds to a detection result, which includes the location, category, and original single-frame detection confidence. The target box refers to the rectangular box used to define the suspected fire area. The location of the target box can be represented by the coordinates of the upper left and lower right corners of the rectangle. The target box category is smoke or open flame.

[0034] For candidate pyrotechnic areas with very low resolution or small size, their location and size are specially marked and recorded so that they can be focused on in subsequent steps.

[0035] S2-2, Optical Flow Field Analysis and Extraction of Key Spatiotemporal Physical Features.

[0036] Taking each candidate fireworks region as the core object, the optical flow field is calculated using data from two or more consecutive frames. The optical flow field describes the direction and velocity of each pixel's motion between adjacent frames, which is the basis for subsequent extraction of motion features.

[0037] Based on the optical flow field, the following five key physical indicators are extracted, which together constitute a set of indicators for characterizing the dynamic physical properties of the target: (1) Regional diffusion index D divergence It describes the degree to which pixels within a candidate region diffuse, expand, or rise outwards. This index is positive when smoke is spreading. Its calculation formula is:

[0038] Where u is the horizontal component of the optical flow field and v is the vertical component; Let u be the spatial difference in the x-direction, representing the degree of horizontal diffusion; It is the spatial difference of v in the y direction; mean() means taking the average value across all pixels in the candidate region.

[0039] (2) Flow field vortex index It is used to detect whether there are vortex flow or entrainment features in the candidate region. This index is significantly non-zero when the smoke is swirling. The calculation formula is:

[0040] in, It is the spatial difference of v in the x direction, approximately v(x+1,y)-v(x,y); It is the spatial difference of u in the y direction, approximately u(x,y+1)-u(x,y).

[0041] (3) Direction index It is used to measure the consistency of pixel motion direction within a candidate region across consecutive frames, and the calculation formula is: = mean(cos( , )) in, The direction of motion of the optical flow vector of a pixel at time t (frame); The direction of the optical flow vector at the same pixel position at time t+1; cos( , ) represents the cosine of the angle between the two directions.

[0042] If all pixels move continuously in the same direction (such as a shifting cloud), C approaches 1; if the directions of movement are chaotic (such as the diffusion of smoke and turbulence), C approaches 0 or is negative.

[0043] (4) Scintillation entropy It is used to reflect the temporal changes in pixel brightness within the candidate region, and is mainly used to capture the flickering and deflagration phenomena of open flames. The calculation formula is:

[0044] in, Let be the brightness value of frame t at position (x, y); The calculation formula is:

[0045] Where I is the pixel brightness value; I t This represents the pixel brightness value of the t-th frame; I t-1 This represents the pixel brightness value of the (t-1)th frame; H(·) represents the histogram distribution of all ΔI within the candidate region and the calculation of information entropy (i.e., the sum of the negative logarithms of the histogram probability distribution).

[0046] (5) Texture blur It is used to distinguish translucent, sparsely defined smoke from clearly defined solid targets or reflections. The calculation formula is as follows:

[0047] Among them, Laplacian (I t To extract edge intensity from candidate region images, the Laplacian operator convolution operation is preferred for calculation. The edge intensity extraction result of the candidate region image reflects the clarity of local edges and textures. The higher the calculated texture blur value, the more blurred and structure-lacking the region is, and the more likely it is to be semi-transparent smoke. The Laplacian operator convolution operation is a second-order differential operator, which is often used for image edge detection.

[0048] It should be noted that the edge intensity extraction method used in this step is not limited to the Laplacian operator; any method that defines texture blur based on the reciprocal of the edge response intensity is also applicable.

[0049] S3, Joint Gated Collaborative Filtering.

[0050] This step aims to jointly determine the five physical indicators mentioned above with the original single-frame detection confidence score. Dynamic gates (motion feature gates) and static gates (appearance feature gates) are set as thresholds. Only when a candidate smoke area passes both thresholds simultaneously will it proceed to the next in-depth analysis; otherwise, it is directly filtered out. Specifically, it includes the following sub-steps: S3-1, Dynamic gate determination.

[0051] A dynamic gate can be defined as a region diffusion index D of a candidate region in consecutive frames.divergence Flow field vortex index Direction of motion index If all of them exceed their respective motion feature thresholds, the candidate pyrotechnic area is determined to have passed the dynamic gate, indicating that its motion pattern conforms to the characteristics of pyrotechnic diffusion, rotation, or drift. The motion feature thresholds corresponding to each indicator group can be actually calibrated by the staff according to the scene.

[0052] S3-2, Static gate determination.

[0053] A static gate can be defined as the scintillation entropy of the candidate region. The texture blur is higher than a lower threshold. If the value is higher than a lower threshold, it is determined to pass the static gate, indicating that its appearance characteristics (brightness changes and texture blur) conform to the visual characteristics of fireworks.

[0054] S3-3, Joint scoring.

[0055] S3-3-1. When a candidate pyrotechnic zone passes through both dynamic and static gates simultaneously, its joint score is calculated using the following formula:

[0056] in, This represents the Sigmoid activation function, used to combine the joint scores. Compress to the [0,1] interval; The original single-frame detection confidence obtained in step S2-1; The weights for the original confidence scores of the target detection model; Regional diffusion index The weighting can be increased if the smoke spreads significantly. ; For consistency of motion direction The weight can be increased if there is significant environmental interference. .

[0057] S3-3-2. After calculating the joint score, if the joint score is higher than the preset threshold, the candidate fireworks area is retained as a candidate target to be fused; otherwise, the target is discarded.

[0058] The above steps effectively filter out meaningless targets such as clouds, reflections, and clutter, thereby reducing the consumption of subsequent computing resources.

[0059] S4, Multi-source feature high-dimensional space coding.

[0060] This step aims to perform high-dimensional spatial encoding on each candidate target obtained in S3, including its original video frame sequence, depth features, and physical index group, to prepare for subsequent multimodal fusion. Please refer to [link to relevant documentation]. Figure 2 The specific sub-steps are as follows: S4-1, High-dimensional encoding of video frame sequences.

[0061] S4-1-1. Collect the continuous video frames containing the candidate target to be fused as video input, denoted as V={v image 1,v image 2,...,v image t}; Among them, v image t ∈H×W×3, where t is the frame index, H / W is the resolution, and 3 is the RGB channel.

[0062] S4-1-2. After scaling, normalizing, and frame sampling the video input V, the preprocessed feature V is obtained. feature .

[0063] V feature The input is the first encoding network, which generates a video frame sequence representation with location information based on a recursive or attention mechanism, and can be represented as follows: (v) feature n F n =w n-1 video (v) feature n-1 F n-1 n=1,2,3…x Among them, w video Information extracted from video frames; F represents the location information of the video in high-dimensional space; n represents the entire sample set; v feature n-1 This represents the (n-1)th video frame.

[0064] S4-2, High-dimensional encoding of optical flow field index vectors.

[0065] The five physical indices obtained in step S2-2 are used to construct a vector L. indicator Vector L indicator It can be represented as: L indicator = ( , , , , ) Subsequently, the vector sequence is input into the second encoding network to generate a physical index representation with location information: (L) indicator n E n =w n-1 indicator (L indicator n-1 E n-1 n=1,2,3…x Among them, w n-1 indicator The transformation function for extracting information from the (n-1)th optical flow field index; E n-1 Encode the position of the (n-1)th optical flow field index in high-dimensional space; L indicator n-1 This represents the (n-1)th optical flow field index.

[0066] S4-3, Video frame depth feature extraction and encoding.

[0067] The same sequence of video frames is input into a convolutional neural network (CNN), which outputs a deep feature map F. feature t It is represented as: F feature t，n =CNN({v image 1,v image 2,...,v image t} n ) The feature map sequence is input into the third encoding network to generate a deep feature representation with location information: (F) feature t，n D n )= w n-1 feature (F feature t,n-1 D n-1 n=1,2,3…x Among them, w feature A transformation function for extracting information from deep features; F feature t,n-1 This represents the feature of the (n-1)th video frame; D n-1 Encodes the position of the (n-1)th deep feature in the high-dimensional space.

[0068] S4-4, Multi-feature fusion coding.

[0069] S4-4-1. To fuse video frame sequence representation, depth feature representation, and physical index representation in the same high-dimensional space, a mapping function is first defined, as follows: Define the feature representation R of the video frame feature sequence in the nth sample. t，n F for: R t，n F = K f n (R) Among them, K f n It is a mapping function for video frame features.

[0070] Define the feature representation R of the video frame sequence in the nth sample. n v for: R n v = K v n (R) Among them, K v n This is a mapping function for video frames.

[0071] S4-4-2. The feature representations of the video frame sequence and the feature representations of the video feature sequence are fused and encoded into a fourth sequence. The fusion process can be represented by the following formula: (h) n , (R t，n F R n v ), O n )= w n-1 mix (h) n-1 , (R t，n-1 F R n-1 v ), O n-1 ) Among them, w n-1 mix Extract and generate the fusion information for the current time step from the fusion state at time n-1; h is the label for the fusion feature. n-1 The label for the (n-1)th fused feature; O n Encode the position information of the nth fused feature in the high-dimensional space; R t，nF This represents the feature sequence of video frames in the nth sample. R n vThis represents the feature representation of the video frame sequence in the nth sample.

[0072] At the same time, the optical flow field index R n L Encoding in high-dimensional space using new labels: R n L = K v n (R) Among them, K v n This represents the same mapping function as the video frame, which is actually applied to the optical flow index L and outputs a high-level integrated representation of the optical flow labeled field: (m) n R n L q n )= u n-1 (m) n-1 R n-1 L q n-1 ) Among them, u n-1 A function for extracting information from n-1 optical flow marker fields; q n This represents the position encoding information of the nth optical flow marker field in high-dimensional space; m n The label is for the nth optical flow label field.

[0073] S4-4-2, The label h of the last fused feature. x (i.e., the fused feature label of the x-th sample) is mapped by the function j w h Projecting onto a high-dimensional space yields the fused feature projection A. w h It is represented as: A w h = j w h (h x ) Label m of the last optical flow field x (i.e., the optical flow label field of the x-th sample) is mapped by the function g. u m Projecting onto a higher-dimensional space yields the optical flow field index characteristic projection B. u m It is represented as: B u m = g u m (mx ) S5. Multimodal spatiotemporal fusion and determination.

[0074] Project the fusion feature obtained in step S4-4 onto A w h Projection of optical flow field index characteristics B u m Simultaneously, the input is fed into a fusion model based on a Transformer network. After processing through multiple layers of the Transformer network, the final fusion features are output. The core of the Transformer network is the self-attention mechanism, which can capture the long-distance dependencies between any two positions within a sequence, as well as the interdependencies between different sequences (video frames, depth features, physical metrics).

[0075] Subsequently, the final fused features are passed sequentially through a fully connected layer (which maps high-dimensional features to the number of categories) and a softmax activation function (which converts the output into a probability distribution) to obtain the final probability that the candidate target to be fused is a real firework.

[0076] If the probability is greater than the preset alarm threshold, it is determined to be real fireworks.

[0077] The above multimodal spatiotemporal fusion process is as follows: Figure 2 As shown.

[0078] S6. Result rendering and alarm output.

[0079] The final detected target bounding boxes, categories, confidence scores, and related metadata (such as timestamps, geographic coordinates, short video clips, etc.) are packaged together.

[0080] The packaged information is sent to the management platform or stored locally through the application programming interface (API) of the software development kit (SDK) to complete the identification and alarm process.

[0081] Figure 3 and Figure 4 The images show typical examples of distant smoke and open flames from a drone's perspective, based on... Figure 3 As can be seen, after adopting this solution, in the drone's distant view image containing building complexes and forests, the area suspected to be smoke was accurately outlined with a red rectangular box in the distant forest, and labeled "general smoke 0.80", indicating that the probability of ordinary smoke in this area is 0.8. And in Figure 4The main scene shown is an image of open flames in farmland. The area of ​​open flames is completely enclosed in orange and labeled "fire 0.81", indicating that there is an open flame in this area with a probability of 0.81. At the same time, the black smoke produced is enclosed in yellow and labeled "black smoke 0.81", indicating that there is black smoke in this area with a probability of 0.81.

[0082] In summary, this embodiment utilizes a combined gating collaborative filtering technique, employing motion feature gates (dynamic gates) and appearance feature gates (static gates) to perform early screening of candidate smoke and fire areas, significantly improving processing efficiency. Simultaneously, by synchronously encoding video frames, depth features, and optical flow physical indicators into a unified high-dimensional space and employing a transform network for multimodal spatiotemporal fusion, it fully learns the dynamic patterns of smoke and fire diffusion, flickering, and rotation over time, effectively suppressing false alarms and missed alarms. This solves the problems of low recognition accuracy and high false alarm rate in existing technologies, achieving a low false alarm rate and a high detection rate even in various complex scenarios (such as backlight, fog, and distant small targets).

[0083] To implement the above method, please refer to [link / reference]. Figure 5 This application also discloses an outdoor smoke and fire intelligent recognition system for high-altitude drone inspection, comprising: The video acquisition and preprocessing module is used to acquire the video stream of the UAV inspection, and to decode and sample key frames of the video stream to obtain the video frame sequence to be analyzed. The preliminary detection module is used to input the video frame sequence into the first deep learning detection network, locate the candidate fireworks area, and generate a preliminary detection result containing the location, category and original confidence of the area; The dynamic physical feature extraction module is used to calculate the optical flow field corresponding to each candidate fireworks region based on the preliminary detection results and the video frame sequence, and extract a set of indicators to characterize the dynamic physical properties of the target based on the optical flow field. The joint gated collaborative filtering module is used to employ a joint gated collaborative filtering method to fuse the original confidence level and the indicator group, calculate the joint score of each candidate fireworks region, filter out candidate fireworks regions with joint scores below a preset threshold as invalid targets, and retain candidate fireworks regions with joint scores above the preset threshold as candidate targets to be fused. The multi-source feature encoding module is used to encode the video frame sequence containing the candidate target to be fused, the deep features extracted by the feature extraction network of the video frame sequence, and the index group in a high-dimensional space to obtain a unified high-dimensional space for video frame representation, deep feature representation and physical index representation. The multimodal spatiotemporal fusion and determination module is used to globally fuse the video frame representation, depth feature representation and physical index representation in time sequence using a multimodal spatiotemporal fusion method to generate fusion features, and calculate and output the final probability that the candidate target to be fused is a real firework based on the fusion features.

[0084] In conjunction with the method of this application, a computer program product is also disclosed, including computer instructions, which, when executed by a processor, specifically perform the steps of the method described in this embodiment.

[0085] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for intelligent identification of outdoor smoke and fire during high-altitude inspection by unmanned aerial vehicles (UAVs), characterized in that, include: The video stream of the drone inspection is acquired, and the video stream is decoded and keyframes are sampled to obtain the video frame sequence to be analyzed. The video frame sequence is input into the first deep learning detection network to locate candidate fireworks areas and generate preliminary detection results including target location, category and original confidence level. Based on the preliminary detection results and the video frame sequence, the optical flow field corresponding to each candidate fireworks area is calculated, and a set of indicators for characterizing the dynamic physical properties of the target is extracted based on the optical flow field. A joint gating collaborative filtering method is adopted, which integrates the original confidence level and the index group to calculate the joint score of each candidate fireworks region. Candidate fireworks regions with joint scores below a preset threshold are filtered as invalid targets, while candidate fireworks regions with joint scores above the preset threshold are retained as candidate targets. The video frame sequence containing the candidate target, the depth features extracted from the video frame sequence by the feature extraction network, and the index group are respectively encoded in high-dimensional space to obtain video frame representation, depth feature representation and physical index representation in a unified high-dimensional space. A multimodal spatiotemporal fusion method is adopted to globally fuse the video frame representation, depth feature representation and physical index representation in time sequence to generate fused features, and the final probability that the candidate target is real fireworks is calculated and output based on the fused features.

2. The method according to claim 1, characterized in that, The set of indicators characterizing the dynamic physical properties of the target is calculated through the following steps: Based on the difference in optical flow components in the horizontal direction and the difference in optical flow components in the vertical direction of the pixels in the candidate region, the sum of the two means in the region is calculated to obtain the regional diffusion index used to characterize the diffusion trend of smoke. Based on the difference between the vertical optical flow component of the pixel in the candidate region and the horizontal optical flow component and the vertical optical flow component, the mean of the two differences is calculated to obtain the flow field vortex index used to characterize smoke vortices and / or entrainment motion. Based on the cosine value of the angle between the optical flow vector of each pixel in the candidate region and the optical flow vector of the previous frame, the mean value of the cosine value in the region is calculated to obtain the motion direction index used to characterize the continuous consistency of the target's motion direction. Based on the brightness change value of the same pixel position in the candidate region between adjacent frames, the histogram distribution of the brightness change value in the region is statistically analyzed and the information entropy is calculated to obtain the flicker entropy used to characterize the flickering of open flames and / or deflagration. Edge intensity is extracted from the image within the candidate region. Based on the average value of the extracted edge intensity, texture blur is obtained to characterize the difference between the texture of the semi-transparent smoke and the texture of the solid target.

3. The method according to claim 2, characterized in that, The joint gating collaborative filtering method includes: A dynamic gate judgment condition is set. When the regional diffusion index, the flow field vortex index, and the motion direction index all exceed their respective motion characteristic thresholds, the candidate pyrotechnic area is determined to pass through the dynamic gate. Set static gate judgment conditions: when the flicker entropy and texture ambiguity meet their respective appearance feature conditions, the candidate fireworks area is determined to pass through the static gate. Only when a candidate pyrotechnic area passes through both a dynamic gate and a static gate simultaneously is its joint score calculated based on a weighted sum of the original confidence level, the area diffusion index, the motion direction index, and the scintillation entropy, and compressed by an activation function; each weight coefficient is adaptively adjusted according to the degree of scene interference.

4. The method according to claim 1, characterized in that, The multimodal spatiotemporal fusion method includes: The video frame sequence containing the candidate target is input into the first coding network and mapped to a high-dimensional space to generate a video frame sequence representation with location information. The vector sequence composed of the index group is input into the second encoding network and mapped to a high-dimensional space to generate a physical index representation with location information. The video frame sequence is input into a convolutional neural network to extract depth feature maps, and the depth feature map sequence is input into a third encoding network to generate a depth feature representation with location information. The video frame sequence representation, depth feature representation, and physical index representation are spliced ​​together in the same high-dimensional space and input into a fusion model based on a transform network to capture the self-dependencies within the sequence and the interdependencies between different representation sequences. The fused features output by the transformation network are sequentially passed through a fully connected layer and a flexible maximum activation function to obtain the final probability that the candidate target is a real firework.

5. The method according to claim 1, characterized in that, After obtaining the video frame representation, depth feature representation, and physical index representation in a unified high-dimensional space, the method also includes result output, which includes: Based on the final probability that the candidate target is a real firework and the preset alarm threshold, metadata including target bounding box, category, confidence level, timestamp and geographic coordinates is generated and reported or / and stored through application programming interface.

6. The method according to claim 2, characterized in that, The diffusion index for the region is calculated using the following method: The mean of the spatial difference values ​​of the horizontal optical flow component and the mean of the spatial difference values ​​of the vertical optical flow component within the candidate region are added together to obtain a value that characterizes the outward expansion and / or diffusion intensity of the target region.

7. The method according to claim 2, characterized in that, The following method is used to calculate the vortex index of the flow field: The spatial difference between the vertical optical flow component and the horizontal optical flow component within the candidate region is subtracted from the spatial difference between the horizontal optical flow component and the vertical optical flow component. The resulting mean is used to characterize whether vortex flow and / or entrainment features exist in the target region.

8. The method according to claim 1, characterized in that, When the first deep learning detection network outputs preliminary detection results, it records the position and size information of candidate fireworks areas with resolution and / or size lower than a preset small target threshold, and uses this information as a key focus when calculating the optical flow field.

9. An intelligent outdoor smoke and fire detection system for high-altitude drone inspection, characterized in that, include: The video acquisition and preprocessing module is used to acquire the video stream of the UAV inspection, and to decode and sample key frames of the video stream to obtain the video frame sequence to be analyzed. The preliminary detection module is used to input the video frame sequence into the first deep learning detection network, locate the candidate fireworks area, and generate a preliminary detection result containing the location, category and original confidence of the area; The dynamic physical feature extraction module is used to calculate the optical flow field corresponding to each candidate fireworks region based on the preliminary detection results and the video frame sequence, and extract a set of indicators to characterize the dynamic physical properties of the target based on the optical flow field. The joint gated collaborative filtering module is used to employ a joint gated collaborative filtering method to fuse the original confidence level and the indicator group, calculate the joint score of each candidate fireworks region, filter out candidate fireworks regions with joint scores below a preset threshold as invalid targets, and retain candidate fireworks regions with joint scores above the preset threshold as candidate targets to be fused. The multi-source feature encoding module is used to encode the video frame sequence containing the candidate target to be fused, the deep features extracted by the feature extraction network of the video frame sequence, and the index group in a high-dimensional space to obtain a unified high-dimensional space for video frame representation, deep feature representation and physical index representation. The multimodal spatiotemporal fusion and determination module is used to globally fuse the video frame representation, depth feature representation and physical index representation in time sequence using a multimodal spatiotemporal fusion method to generate fusion features, and calculate and output the final probability that the candidate target to be fused is a real firework based on the fusion features.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they specifically perform the steps in the method as described in any one of claims 1-8.

Images