A method for detecting and analyzing fireworks based on YOLO algorithm and dynamic analysis

Abstract

The present invention belongs to the field of data processing technology, and specifically relates to a method for detecting and analyzing fireworks based on a YOLO algorithm and dynamic analysis, the method comprising: step 1: using a trained fireworks detection model to detect fireworks candidate targets in a target video and record their coordinate positions as a target detection frame set; step 2: using a method based on inter-frame difference to determine whether there is a moving object in the current frame of the target video, and calculating the intersection and union ratio of the moving object and the target detection frame set as the abnormality ratio; step 3: if the abnormality ratio does not exceed a set abnormality discrimination threshold, if it is determined to be nonlinear motion, an alarm message is output, if it is determined to be linear motion, the record is cleared, and no alarm message is output. The present invention significantly improves the accuracy, real-time and robustness of fireworks detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a smoke and fire detection and analysis method based on a YOLO algorithm and dynamic analysis. Background Art

[0002] With the continuous development of society, fire monitoring and early warning systems have become particularly important in the fields of urban safety, industrial production and public place management. The harm caused by fire is huge. How to detect fire quickly and accurately in the early stage and issue an alarm in time has become a hot topic of research. The existing fire detection technology is mainly divided into traditional smoke and flame sensor detection technology, fire detection technology based on visual analysis, and multi-sensor fusion-based methods. These technologies have their own advantages and disadvantages, but there are still some problems in practical applications, especially when detecting fire targets in real time in complex environments and large-scale scenes, the effect is often not ideal.

[0003] Traditional fire detection methods mainly rely on physical means such as smoke and temperature sensors to detect fire characteristics. This type of technology has certain advantages, such as being sensitive to temperature and smoke. However, these sensors often have high requirements for the detection environment and are easily affected by factors such as ambient temperature, humidity, and air circulation, resulting in high false alarm and missed alarm rates. In addition, sensor detection usually requires a certain amount of propagation time. For example, it may take from several seconds to tens of seconds for smoke to spread to the location of the sensor. This seriously affects the real-time nature of fire detection, especially in the early stages of a fire, when it is often difficult to detect the fire in time and respond quickly.

[0004] In the development of fire detection technology, fire detection methods based on computer vision have gradually become an important means. It analyzes and processes image data in surveillance videos to detect the characteristics of fire targets such as flames and smoke to achieve early warning. Typical visual fire detection methods include flame recognition based on color features, smoke detection based on motion features, and target detection methods combined with deep learning. The color feature-based method mainly uses the color features of flames, such as red, orange, and yellow, to identify the flame area through a specific color model; the motion feature-based method uses the dynamic characteristics of smoke such as diffusion and drift, combined with optical flow, inter-frame difference and other means to detect the movement of smoke. However, these feature-based detection methods are usually affected by ambient lighting, background interference and camera shooting angles, and the detection effect is unstable and prone to false alarms. Summary of the invention

[0005] In view of this, the main purpose of the present invention is to provide a smoke and fire detection and analysis method based on the YOLO algorithm and dynamic analysis, which significantly improves the accuracy, real-time and robustness of smoke and fire detection.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A method for detecting and analyzing fireworks based on the YOLO algorithm and dynamic analysis, the method comprising:

[0008] Step 1: Use the trained fireworks detection model to detect fireworks candidate targets in the target video and record their coordinate positions as the target detection box set;

[0009] Step 2: Use the frame difference-based method to determine whether there is a moving object in the current frame of the target video, and calculate the intersection-and-union ratio of the moving object and the target detection frame set as the abnormality ratio. If the abnormality ratio exceeds the set abnormality discrimination threshold, start timing and mark this frame as abnormal, otherwise mark this frame as normal, and repeat the cycle until the predetermined time is met;

[0010] Step 3: If the abnormality ratio does not exceed the set abnormality judgment threshold, the first recorded frame will be released, and then the newly added frame information will be recorded. If there is no abnormal frame record for a set number of consecutive frames, the recorded frame information will be cleared; if the abnormality ratio meets the requirements, the recorded abnormal frame information will be judged for linear motion. If it is judged to be non-linear motion, an alarm message will be output. If it is judged to be linear motion, the record will be cleared and no alarm message will be output.

[0011] Furthermore, the fireworks detection model is YOLOv8s.

[0012] Furthermore, step 1 specifically includes: for each frame of the target video Perform preprocessing, adjust the image size to the target value and normalize it; Represents the index of the time frame, which indicates the first The image of the frame is in the range of ,in is the total number of frames of the target video; the pre-trained fireworks detection model is used to infer each frame of the pre-processed image to obtain the position and confidence of the detection frame; the fireworks detection model uses a neural network to divide each frame into Grids, each grid predicts multiple bounding boxes and corresponding confidence and category probabilities; the output of the fireworks detection model is represented as a three-dimensional tensor: ;in, is the output tensor containing the prediction information for each grid; is the number of grid divisions; The number of bounding boxes predicted for each grid; Indicates that each bounding box contains 5 parameters: center coordinates ,width ,high And confidence ; Indicates the X-axis coordinate; Indicates the Y-axis coordinate; is the number of categories, indicating the probability of the category of fireworks; the bounding boxes output by the model are filtered, and the confidence level is higher than the threshold. The bounding box of the image is used as the candidate box; the coordinates of the candidate box are adjusted from the relative grid coordinates to the absolute image coordinates; non-maximum suppression is performed on the adjusted candidate box to remove duplicate detection boxes; for each adjusted candidate box, the intersection and union ratio with other boxes is calculated, and if the intersection and union ratio is higher than the set threshold , then the candidate box with the highest confidence is retained to form the target detection box set; Represents the field of real numbers.

[0013] Furthermore, let the size of each frame of the target video be , resize it to the target size , and then normalize it; let the horizontal direction , vertically The grid The confidence of the predicted bounding box is expressed as ; The bounding box output by the model is filtered using the following formula:

[0014] ;

[0015] Among them, in the mesh division of the smoke and fire detection model, and Represents the position index of the grid. The fireworks detection model divides each frame into The grid, and Respectively represent the index position of the grid in the vertical and horizontal directions; the ranges are and ; Represents the index of the predicted bounding box in each grid; the fire detection model predicts multiple bounding boxes for each grid to capture different target candidate boxes; The range is ,in is the number of bounding boxes predicted for each grid; is the set of candidate boxes after filtering; The horizontal direction after filtering , vertically The grid The coordinates and sizes of the candidate boxes corresponding to the predicted bounding boxes, is the corresponding X-axis coordinate, is the corresponding Y-axis coordinate, is the corresponding width; The corresponding height.

[0016] Furthermore, the candidate box coordinates are converted from relative grid coordinates to absolute image coordinates using the following formula:

[0017] ;

[0018] in: and are the center coordinates of the X axis and the center coordinates of the Y axis of the absolute image coordinates respectively; and are the width and height of the candidate box coordinates adjusted to absolute image coordinates; Represents the Sigmoid function, which is used to limit the center coordinates to within the scope; is the scale adjustment coefficient, which is used to control the nonlinear change of the bounding box size.

[0019] Furthermore, the following formula is used to perform non-maximum suppression on the adjusted candidate boxes to remove duplicate detection boxes:

[0020] ;

[0021] if ; then remove ;in: Candidate box With candidate box The intersection and union ratio of is the overlapping area of ​​the two candidate boxes; is the total area of ​​the two candidate boxes; and All are subscript indexes.

[0022] Furthermore, in step 2, a method based on inter-frame difference is used to determine whether there is a moving object in the current frame of the target video, and the intersection and union ratio of the moving object and the target detection frame set is calculated as the abnormality ratio. The specific process includes: and the previous frame , calculate the gradient difference ; Introduce a multi-threshold adaptive strategy to calculate the local threshold according to the gradient characteristics of different regions and the nonlinear enhancement coefficient , for gradient difference Perform binarization processing to obtain the binarization result ; Binarization result Connected domain detection is performed to identify continuous motion areas. The intersection-and-union ratio of the identified motion area and the set of target detection frames is calculated. If the intersection-and-union ratio of the motion area and the set of target detection frames exceeds the set threshold, the abnormality ratio is calculated and the current frame is marked.

[0023] Further, use the following formula to calculate the current frame and the previous frame , calculated in different directions and different scales Gradient difference on :

[0024] ;

[0025] in, is the gradient difference, which means in coordinates The difference value at ; For the direction The Gaussian filter on , with a scale parameter of ; Indicates in the direction Gradient calculation on ; Represents the convolution operation; is the nonlinear enhancement coefficient, which controls the degree of enhancement of the difference value; the binarization result is obtained through the following formula: :

[0026] ;

[0027] in, , is the center of the grid The neighborhood of the center The average difference value of is the local dynamic smoothing coefficient, ; is the set spatiotemporal smoothing parameter; is the nonlinear mapping coefficient; is the X-axis coordinate of the center of the grid; The Y-coordinate of the center of the grid.

[0028] Furthermore, the process of calculating the abnormality ratio specifically includes: For the detected connected domain set, calculate the weighted area of ​​each connected domain :

[0029] ;

[0030] in, Connected domain The weighted area of ​​? is a nonlinear weight, determined according to the image gradient change; is the number of connected domains; let the target detection box set , then calculate the intersection and union ratio of the identified motion area and the target detection frame set :

[0031] ;

[0032] in, is the number of candidate boxes in the target detection box set; Represents the first Candidate boxes; is the first Connected domains; calculate the anomaly ratio using the following formula: :

[0033] .

[0034] By adopting the above technical scheme, the present invention produces the following beneficial effects: the present invention combines the methods of inter-frame difference and dynamic behavior analysis, making up for the deficiency of relying solely on static image detection. By calculating the gradient difference between the current frame and the previous frame, the moving object in the video is detected, and the abnormal proportion is accurately determined by combining the intersection-and-union ratio of the target detection frame. This dynamic analysis method can effectively capture the motion characteristics of fireworks targets, such as the jumping of flames and the diffusion of smoke, and improve the sensitivity of detection. At the same time, by using multi-threshold adaptive strategy and nonlinear enhancement processing, the system can adapt to complex environmental changes in different scenes, and reduce the influence of factors such as illumination changes and background interference on the detection results. The present invention optimizes the accuracy and effectiveness of the target detection frame by introducing non-maximum suppression (NMS) and coordinate transformation of the candidate frame. Non-maximum suppression effectively removes duplicate detection frames, ensuring that each fireworks target corresponds to only one high-confidence bounding box, reducing redundant information. At the same time, the candidate frame is converted from relative grid coordinates to absolute image coordinates, ensuring that the position of the detection frame is accurately mapped to the original image, and improving the accuracy of positioning. These optimization measures enable the system to maintain a high level of detection performance in complex scenes. The present invention realizes high-precision real-time detection of fireworks targets by introducing advanced target detection algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A method flow chart of a method for smoke and fire detection and analysis based on the YOLO algorithm and dynamic analysis provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any manner.

[0037] Any feature disclosed in this specification (including any additional claims and abstract), unless otherwise stated, may be replaced by other alternative features that are equivalent or have similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

[0038] Example 1: Reference Figure 1 , a method for detecting and analyzing fireworks based on the YOLO algorithm and dynamic analysis, the method comprising:

[0039] Step 1: Use the trained fireworks detection model to detect fireworks candidate targets in the target video and record their coordinate positions as the target detection box set;

[0040] Specifically, the fireworks detection model adopts the YOLO model, which is a real-time target detection algorithm based on deep learning. It can quickly and accurately detect the fireworks targets that may appear in the video frame, and form a detection frame set by marking the coordinates, laying the foundation for subsequent dynamic analysis. In this invention, the characteristics of fireworks are first pre-trained by a deep learning model. The pre-training model is based on a large amount of fireworks pictures and video data, so that the model can learn the characteristics of fireworks in different scenes, including the shape, color, edge characteristics, brightness changes of the flame, and the diffusion, diffusion form and motion pattern of the smoke. The advantage of the YOLO algorithm lies in its end-to-end target detection capability, that is, after inputting a complete frame image, the algorithm divides the image into S×S grids, and then predicts the target category and position coordinates in each grid through a convolutional neural network to form the so-called candidate target detection frame. In this process, the YOLO model can detect multiple fireworks targets in the video frame at one time, and generate a target detection frame set containing bounding box coordinates, confidence, and category probability. Compared with traditional sliding window detection or two-stage detection methods, such as R-CNN and its variants FastR-CNN, FasterR-CNN, etc., the YOLO algorithm can complete target detection and classification through a single forward propagation, without the need for multiple region proposals and classifications, thereby greatly improving the detection speed and real-time performance.

[0041] The target detection capability of the YOLO model is particularly significant for fireworks detection. Fireworks are highly dynamic. Flames may undergo dramatic changes in shape and brightness in a short period of time. The boundaries of smoke are unclear and the diffusion speed is uneven. Conventional target detection algorithms are prone to false detection or missed detection due to these complex characteristics. However, the global detection strategy adopted by the YOLO algorithm, that is, learning the overall characteristics of fireworks targets through full-image information rather than matching local feature points, is robust to the detection of targets with complex shapes and rapid changes. This global feature extraction and detection method enables YOLO to capture the edge shape and brightness gradient of flames well, and model the diffusion and transparency of smoke, thereby generating accurate target detection frames. In addition, the confidence prediction mechanism of the YOLO model for targets can quantify the probability of fireworks targets being correctly identified. For those targets that are suspected of being fireworks but are uncertain, the model can screen them through confidence thresholds to ensure the accuracy of the detection results. On the other hand, the YOLO model uses non-maximum suppression (NMS) to avoid overlapping of multiple detection frames during the generation of detection frames. For adjacent or partially overlapping fireworks targets in the video frame, NMS can retain the most representative detection frame to ensure the simplicity and accuracy of the detection frame set. In this invention, the fireworks detection frame set generated by the YOLO algorithm will be used for dynamic analysis in subsequent steps. Therefore, the accuracy and real-time performance of target detection directly affect the performance of the entire system. By using the pre-trained YOLO model to detect fireworks targets, the area suspected of fireworks in the video frame can be quickly locked and coordinate annotations can be formed, which not only reduces the amount of calculation for subsequent processing, but also improves the real-time performance of the entire system. In practical applications, the fireworks detection frame set will become the input of dynamic analysis to further judge the motion characteristics and abnormal behavior of the target, and ultimately achieve accurate recognition and early warning of fireworks.

[0042] Step 2: Use the frame difference-based method to determine whether there is a moving object in the current frame of the target video, and calculate the intersection-and-union ratio of the moving object and the target detection frame set as the abnormality ratio. If the abnormality ratio exceeds the set abnormality discrimination threshold, start timing and mark this frame as abnormal, otherwise mark this frame as normal, and repeat the cycle until the predetermined time is met;

[0043] Inter-frame difference is a commonly used motion detection method, and its principle is to judge whether there is a moving object in the image by comparing the pixel difference between two or more consecutive frames in the video. Under normal circumstances, under a stable background, the pixel difference between adjacent frames is small, and when an object moves, the pixel value of the moving area will change greatly, so that the moving area can be detected by inter-frame difference. In the present invention, this inter-frame difference technology is used to detect moving targets in the video stream, so as to judge whether the current frame contains a moving object. The method calculates the pixel difference map between adjacent frames, sets a certain threshold to identify the moving area, and marks the moving object. Specifically, the difference map obtained by inter-frame difference will highlight the pixels that have changed in the current frame, so that the moving object can be effectively distinguished from the static background. In the fireworks detection scene, flames and smoke often spread and move at a high speed and in an irregular manner. These characteristics make them very obvious in the inter-frame difference map, so that they are easier to be identified. After detecting the moving object in the video, the present invention further calculates the intersection and union ratio of the moving object and the fireworks candidate target detection frame previously detected by the YOLO algorithm. The intersection-and-union ratio is an indicator that measures the degree of overlap between two areas, and is defined as the ratio of the intersection area to the union area of ​​the two areas. In the present invention, the intersection-and-union ratio is used to measure the degree of spatial overlap between the currently detected moving object and the fireworks target detection frame. The reason why the intersection-and-union ratio is used for calculation is that fireworks targets usually have obvious motion characteristics. The higher the degree of overlap between its moving area and the fireworks detection frame, the more likely it is that the moving object is caused by fireworks, thereby improving the accuracy of detection. In practical applications, by calculating the intersection-and-union ratio between the moving object and each target detection frame, those moving objects that may be related to the fireworks target can be screened out.

[0044] Furthermore, in order to judge the abnormal degree of the current frame in the video, the present invention introduces the concept of abnormal proportion. The abnormal proportion is determined by calculating the intersection-and-union ratio between the moving object and the target detection frame, which reflects the overlap degree of the detected motion area in the target detection frame set. The calculation of the abnormal proportion helps to judge whether the current frame has a high probability of abnormality, that is, whether the activity of the moving object in the frame is consistent with the characteristics of the fireworks target. When the abnormal proportion exceeds the set abnormal discrimination threshold, it means that the overlapping degree of the moving object and the fireworks target in the current frame is high, and there is a high possibility that the abnormal motion is caused by fireworks. In this case, the system will start timing the frame and mark it as an abnormal frame, which means that the frame is likely to contain fireworks or related dangerous behaviors. At the same time, in order to improve the system's ability to distinguish normal and abnormal frames, the present invention sets a predetermined time loop mechanism in the process of detecting moving objects and calculating the intersection-and-union ratio, that is, each frame in the target video is cyclically detected until a certain time length is met. The cyclic detection mechanism can ensure that the system fully analyzes the motion state in continuous frames to avoid misjudgment caused by single-frame noise or accidental situations. At the same time, the predetermined time length can also be used to measure the duration of abnormal motion. For those short non-firework motions, the time threshold can be used to filter them, thereby further reducing false alarms. Through this time dimension judgment, the system can achieve more efficient and accurate firework detection in dynamic video scenes.

[0045] Step 3: If the abnormality ratio does not exceed the set abnormality judgment threshold, the first recorded frame will be released, and then the newly added frame information will be recorded. If there is no abnormal frame record for a set number of consecutive frames, the recorded frame information will be cleared; if the abnormality ratio meets the requirements, the recorded abnormal frame information will be judged for linear motion. If it is judged to be non-linear motion, an alarm message will be output. If it is judged to be linear motion, the record will be cleared and no alarm message will be output.

[0046] Specifically, in step 2, by calculating the intersection and union ratio of the moving object and the target detection frame, the system can preliminarily determine whether there is possible abnormal fireworks behavior in the current frame. However, since the fireworks targets in actual scenes have highly dynamic change characteristics and are usually accompanied by irregular motion patterns, such as the jumping of flames, the diffusion of smoke, etc., these characteristics may have certain similarities with the motion of other non-fireworks objects, so the detection of a single frame alone may lead to false alarms. In order to improve the accuracy of fireworks detection, the present invention introduces a continuous frame analysis method, that is, by analyzing the motion characteristics of the frame sequence in the time dimension, the authenticity of the anomaly can be further determined.

[0047] Specifically, when the abnormal proportion does not reach the set abnormal discrimination threshold, the system will release the first recorded frame and process the newly entered frame. This operation ensures that the system can update the frame information in real time and focus on the latest video screen, so as to ensure that the monitoring of fireworks movement is dynamic and continuous. At the same time, the system will continue to monitor the motion state of the current frame. If no abnormal frame is detected for a set number of consecutive frames, all previously recorded frame information will be cleared. The design of this clearing mechanism can effectively reduce the false alarm rate and avoid the system from mistakenly identifying fireworks targets due to accidental movement. When a series of frames are in a normal state, the system can assume that there is no actual fireworks in the current scene. This automatic update and clearing strategy of frame information makes the system more robust in real-time detection. When the abnormal proportion reaches the set threshold, the system believes that there is a high probability of detecting fireworks or related abnormal behaviors, so the information of the abnormal frame will be recorded and further motion feature analysis will be performed. In this process, the present invention adopts a strategy for distinguishing linear motion and nonlinear motion as a criterion for judging the nature of the abnormality. The motion characteristics of fireworks are usually nonlinear, that is, the flames will jump, flicker, and rotate randomly, and the smoke will diffuse and drift in an irregular form; in contrast, linear motion usually represents a regular and continuous motion trajectory, such as a vehicle driving, a person walking, etc. In the video frame sequence, by analyzing the target motion trajectory, it can be determined whether the target's motion presents a linear and continuous characteristic, or an irregular, dynamically changing nonlinear characteristic.

[0048] In order to realize the discrimination of this motion feature, the system will perform trajectory analysis on the recorded abnormal frames when the abnormal proportion reaches the threshold value to determine whether its motion trajectory meets the characteristics of linear motion. If the judgment result shows that the abnormal frame presents linear motion, the system will consider that this is not an abnormality caused by fireworks, so it will not trigger an alarm, but will clear the recorded abnormal frame information; if the trajectory analysis shows that the motion of the abnormal frame is nonlinear, the system will consider that there is a greater risk of fireworks, and thus output an alarm message to remind relevant personnel to take necessary preventive measures. This analysis method based on motion features enables the present invention to have higher accuracy and pertinence when detecting fireworks targets. Traditional fireworks detection algorithms can often only identify targets based on static features, and have limited analysis capabilities for motion states, so they are prone to false alarms or missed alarms in complex scenarios. The present invention can not only detect fireworks targets, but also effectively distinguish abnormalities caused by fireworks from other non-fireworks movements through the comprehensive application of inter-frame differences, intersection-over-combination ratios, and motion trajectory analysis. In practical applications, this method can significantly improve the performance of fireworks detection systems in dynamic scenes, especially in environments where target features change rapidly and the background is complex. The system can quickly identify fireworks and issue alarms in a timely manner.

[0049] Furthermore, the motion feature analysis method of the present invention also takes into account the persistence and burst characteristics of fireworks targets. Flames and smoke usually show strong bursts in the initial stage, and then gradually spread and stabilize. Therefore, when judging abnormal frames, it is necessary not only to pay attention to the motion trajectory of the target, but also to consider its temporal persistence and morphological changes. By comprehensively analyzing the temporal and spatial characteristics of continuous abnormal frames, the present invention can effectively identify the different states of fireworks targets, such as judging the burning degree of the flame, the diffusion range of the smoke, etc., so as to achieve more accurate detection and analysis.

[0050] After detecting an abnormal target in a video frame, in order to determine whether its motion presents linear characteristics, it is first necessary to extract the motion trajectory of the abnormal target in consecutive frames. The trajectory extraction method is to calculate the coordinate change of the target center point based on the position of the target detection box in multiple frames to form time series data. For example, assuming that in the An abnormal target is detected in the frame, and its position coordinates are , then the target's motion trajectory can be expressed as a time series ,in is the frame number, and are the horizontal and vertical coordinates of the target in the frame. In consecutive frames, the complete motion trajectory of the target can be obtained by tracking the position changes of the target detection frame. In order to analyze the extracted trajectory more accurately, the present invention will smooth the trajectory data. Since there may be noise or target jitter in the video, resulting in discontinuous or non-smooth trajectories, smoothing can reduce errors and extract the main trends of target motion. Common smoothing methods include using Kalman filtering, mean filtering or low-pass filtering to smooth the coordinate sequence. The smoothed trajectory can better reflect the actual motion characteristics of the target and avoid being disturbed by noise.

[0051] In order to determine whether the target's motion trajectory presents linear characteristics, the present invention will perform linear fitting on the smoothed trajectory data. The linear fitting process is to transform the target's coordinate sequence Fit a straight line to evaluate whether the target's motion trend is approximately linear motion. Linear fitting can be performed using the least squares method to obtain the parametric equation of the fitted line, as shown below:

[0052] ;

[0053] in, is the slope of the fitted straight line, is the intercept. The goal of the fitting process is to find a straight line so that the distance (i.e., residual) between the target's motion trajectory point and this straight line is minimized. After the linear fitting is completed, the residual from the trajectory point to the fitting straight line (i.e., the vertical distance from the point to the straight line) is calculated, and the sum of the squares of the residuals is usually used to measure the quality of the fit. In order to determine whether the target's motion trajectory is a linear motion, the present invention introduces a linear discrimination criterion. Normally, if the target's motion trajectory presents a regular straight line motion, then the residual value should be small, and the trajectory points should be evenly distributed near the fitting straight line. Therefore, the residual threshold can be set To determine whether the target motion is linear:

[0054] if , the target motion is considered to be linear;

[0055] If the residual sum of squares of the fit is less than the preset threshold , then it is considered that the target's motion conforms to the linear characteristics; on the contrary, if the residual is large, it means that the trajectory deviates from the linear motion to a large extent, and the target's motion may present nonlinear characteristics, such as irregular jumping, rotation or curvilinear motion. Considering that the target's motion trajectory may present different characteristics in different time periods, the present invention can perform time window analysis on the target's motion trajectory, that is, divide the trajectory into segments, and perform linear fitting and judgment in each segment. This multi-segment linear analysis method can better capture the motion state of the target at different time points, and avoid overall misjudgment due to misjudgment of a single segment trajectory. For example, the target may show linear motion in a period of time, but may suddenly change direction in another period of time. The overall motion characteristics of the target can be more accurately judged through segmented analysis. In addition to analyzing the spatial trajectory of the target, the present invention also combines the speed and direction characteristics of the target for linear discrimination. Linear motion targets usually have constant speed and direction, while nonlinear motion is manifested as random changes in speed and direction. Therefore, the speed of the target in each frame can be calculated. The system can determine the stability of speed and direction by looking at the angle of change in speed and direction. If the target's speed and direction change in the entire trajectory is small and presents linear characteristics, it can be further confirmed that the target's motion is linear. Based on the above analysis, if the target's motion is determined to be linear, the system will assume that the anomaly is not caused by fireworks and will clear the recorded abnormal frame information to avoid false alarms; if it is determined to be nonlinear motion, the system will output an alarm message to indicate that there may be fireworks or other abnormal conditions. This alarm mechanism ensures that the system can accurately identify fireworks targets, avoid false alarms, and improve the accuracy and real-time performance of detection.

[0056] Embodiment 2: The fire detection model is YOLOv8s.

[0057] Specifically, in YOLOv8s, the model extracts multi-scale features from the input image and performs target detection at different feature layers. The model introduces an adaptive anchor mechanism that can better locate fireworks targets, especially for irregular-shaped targets such as flames and smoke. The anchor mechanism can adjust the size of the detection box according to the shape and position of the target to improve the detection accuracy. Due to the lightweight design and efficient reasoning mechanism of YOLOv8s, the model can achieve real-time target detection with low computing resources. This feature is particularly critical in fireworks detection scenarios because it is necessary to respond quickly to changes in flames and smoke and output alarm information in a timely manner. The reasoning speed and lightweight model of YOLOv8s make it very suitable for deployment in scenarios with high real-time requirements such as embedded devices, drones, and monitoring systems.

[0058] Embodiment 3: Step 1 specifically includes: for each frame of the target video Perform preprocessing, adjust the image size to the target value and normalize it; Represents the index of the time frame, which indicates the first The image of the frame is in the range of ,in is the total number of frames of the target video; the pre-trained fireworks detection model is used to infer each frame of the pre-processed image to obtain the position and confidence of the detection frame; the fireworks detection model uses a neural network to divide each frame into Grids, each grid predicts multiple bounding boxes and corresponding confidence and category probabilities; the output of the fireworks detection model is represented as a three-dimensional tensor: ;in, is the output tensor containing the prediction information for each grid; is the number of grid divisions; The number of bounding boxes predicted for each grid; Indicates that each bounding box contains 5 parameters: center coordinates ,width ,high And confidence ; Indicates the X-axis coordinate; Indicates the Y-axis coordinate; is the number of categories, indicating the probability of fireworks categories; the bounding boxes output by the model are filtered and the ones with confidence levels higher than the threshold are retained The bounding box of the image is used as the candidate box; the coordinates of the candidate box are adjusted from the relative grid coordinates to the absolute image coordinates; non-maximum suppression is performed on the adjusted candidate box to remove duplicate detection boxes; for each adjusted candidate box, the intersection and union ratio with other boxes is calculated, and if the intersection and union ratio is higher than the set threshold , then the candidate box with the highest confidence is retained to form the target detection box set; Represents the field of real numbers.

[0059] Specifically, each frame of the video Before being sent to the detection model, they need to go through a series of preprocessing operations. This is because the frames in the video may be of different sizes and resolutions, and the fireworks detection model requires standardized input. Therefore, adjusting the image to the target size can not only improve the consistency of model reasoning, but also ensure the accuracy of the detection results. In addition, the normalization of the image is also to ensure that the data is propagated at an appropriate scale in the network, avoiding problems such as gradient disappearance or explosion, thereby improving the convergence speed and prediction performance of the model. The fireworks detection model is an object detection model optimized for real-time and detection accuracy. It processes images by dividing the image into The smoke and fire detection model adopts a lightweight design in the network structure, and optimizes the efficiency of feature extraction, prediction and reasoning, so that it can achieve real-time detection while maintaining high accuracy. Each grid predicts multiple bounding boxes and provides 5 key parameters for these bounding boxes: center coordinates ,width ,high , and the confidence At the same time, the model also predicts the probability distribution of each bounding box belonging to different categories (such as fire, smoke), and this information is combined into an output tensor , whose shape is When the model infers each frame, it generates multiple candidate boxes based on the size of the grid and the number of predicted bounding boxes. These candidate boxes contain a lot of information, including the location information of the target and the confidence of the target's existence.

[0060] However, not all predicted bounding boxes have high confidence, and not every bounding box contains the actual fireworks target. In order to improve the accuracy of detection, the model screens these candidate boxes and retains those with confidence higher than the set threshold. This threshold The setting of is crucial, which directly affects the sensitivity and accuracy of detection. Setting a higher threshold can reduce false alarms, but may lead to missed detections; a lower threshold will retain more bounding boxes, but may lead to false alarms. Therefore, in practical applications, the threshold needs to be adjusted according to the importance of the scene and the detection target. After the candidate boxes are screened, there may still be multiple overlapping bounding boxes because the same target may be detected by multiple grids. In order to remove these redundant bounding boxes, Example 3 converts the coordinates of the candidate boxes from relative grid coordinates to absolute image coordinates, and then performs non-maximum suppression (NMS). The core idea of ​​non-maximum suppression is to calculate the intersection over union (IoU) between bounding boxes, that is, the ratio of the area of ​​the overlapping area of ​​the two boxes to the area of ​​the union. If the intersection over union of the two bounding boxes is higher than the set threshold , they are considered to be repeated detections of the same target, and only the frame with the highest confidence is retained. Through such optimization processing, it can be ensured that the final detection frame set only contains high-confidence, non-redundant fireworks targets. After non-maximum suppression, the obtained target detection frame set accurately marks the locations where fireworks targets may exist in the video frame. The target detection frame information at this time will provide a basis for subsequent dynamic analysis, including the steps of motion trajectory analysis, behavior recognition and alarm generation of fireworks targets. It is worth noting that the fireworks detection model of the present invention can well cope with complex situations in fireworks scenes, such as irregular flame shapes, diverse color changes, and the diffusion and dispersion characteristics of smoke. The diversity of these targets is fully handled in the grid prediction and bounding box screening mechanism of the model, so that the detection frame not only has high confidence, but also has accurate position and shape matching.

[0061] Example 4: Assume that the size of each frame of the target video is , resize it to the target size , and then normalize it; let the horizontal direction , vertically The grid The confidence of the predicted bounding box is expressed as ; The bounding box output by the model is filtered using the following formula:

[0062] ;

[0063] Among them, in the mesh division of the smoke and fire detection model, and Represents the position index of the grid. The fireworks detection model divides each frame into The grid, and Respectively represent the index position of the grid in the vertical and horizontal directions; the ranges are and ; Represents the index of the predicted bounding box in each grid; the fire detection model predicts multiple bounding boxes for each grid to capture different target candidate boxes; The range is ,in is the number of bounding boxes predicted for each grid; is the set of candidate boxes after filtering; The horizontal direction after filtering , vertically The grid The coordinates and sizes of the candidate boxes corresponding to the predicted bounding boxes, is the corresponding X-axis coordinate, is the corresponding Y-axis coordinate, is the corresponding width; The corresponding height.

[0064] Specifically, the image size of each frame of the target video is set to , this initial size is usually not uniform and does not necessarily meet the input requirements of the detection model. Therefore, each frame image is resized to a standardized size , which facilitates the unified processing of the neural network model. The purpose of resizing is not only to adapt the model structure, but also to ensure that the model has a similar perceptual scale in different scenes, thereby reducing the detection error caused by different image sizes. In addition, the resized image is normalized to standardize its pixel values ​​to a certain range (usually or ). Normalization helps speed up the model training process and ensures that the input features of each layer in the feedforward neural network remain stable, avoiding the situation where the values ​​are too large or too small, causing the gradient to disappear or explode. This can improve the stability of the model during feature extraction and reasoning, making the fireworks detection model have good adaptability to various image inputs.

[0065] The YOLO model uses a "single-step detection" method to predict the target image. One of its major features is to divide the image into In the fire and smoke detection of the present invention, each grid is responsible for detecting possible targets (flames or smoke) in the area and estimating their positions and confidence levels. Specifically, the grid indexes in the horizontal and vertical directions are respectively and Indicates that and , which means that the entire image is evenly divided into Each grid is responsible for generating a set of bounding boxes, which are used to locate possible target areas. For each grid, the model predicts multiple bounding boxes. Represents the predicted bounding box index in each grid, ranging from ,in Indicates the number of bounding boxes predicted for each grid. This process ensures that each grid can capture possible targets in the local area, especially for targets with irregular shapes and fast changes such as flames and smoke. The prediction of multiple bounding boxes can improve the robustness of detection. The prediction information of each bounding box consists of coordinates and confidence. Confidence Indicates the probability that the model believes that there is an object in the current bounding box. This is an important screening indicator because during the detection process, the model will predict a large number of grids and bounding boxes in the image, but not all bounding boxes have high confidence. Therefore, the core of the screening process is to set the confidence threshold , retain bounding boxes with high confidence as candidate boxes, and ignore those bounding boxes with low confidence that are unlikely to contain objects.

[0066] Since the object detection model uses relative coordinates in the grid division, the position of the bounding box needs to be converted from the relative coordinates of the grid to the absolute image coordinates. This conversion process is to ensure that the coordinates of the bounding box can be accurately mapped to the overall size of the image. Since the size of each grid is fixed, the coordinates of the candidate box can be mapped to the absolute image coordinates by multiplying the relative coordinates with the grid size, so that the position of the bounding box is accurately represented in the entire image. Contains all confidence levels exceeding the threshold However, since the model may predict multiple bounding boxes in each grid, and these bounding boxes may overlap or even completely contain each other in space, directly using these bounding boxes may lead to multiple detections of the target. To avoid this, Example 4 further performs non-maximum suppression (NMS) after candidate box screening to eliminate duplicate bounding boxes. The core idea of ​​NMS is to calculate the intersection over union (IoU) between candidate boxes, that is, the ratio of the area of ​​the overlapping area of ​​two bounding boxes to the area of ​​the union. For an IoU higher than a set threshold Only the box with the highest confidence is retained, and other boxes with lower confidence are deleted. This processing method can effectively reduce the redundancy of candidate boxes and ensure that the final set of detection boxes is Each box in the graph represents a single target.

[0067] NMS can handle the characteristics of flames and smoke in fireworks detection scenarios well, because these targets often present irregular shapes and varying sizes. In complex backgrounds, there may be multiple bounding boxes that capture part or all of the fireworks targets, but these bounding boxes may overlap significantly. By retaining the boxes with the highest confidence by NMS, the system can accurately locate the target and avoid false positives due to multiple detections. This processing method is particularly effective when detecting multiple targets, especially when the boundaries of flames and smoke are blurred and the distance between targets is close, NMS can help the system focus on the most important target areas. Throughout the screening and optimization process, the coordinates and size parameters of the candidate boxes are precisely processed and adjusted to ensure that each bounding box can accurately reflect the actual position and size of the target in the image. and Indicates that the center of the bounding box is and The absolute coordinate position on the axis, and and Represents the width and height of the bounding box. After conversion and optimization, these parameters will be used for further fireworks dynamic analysis and behavior recognition.

[0068] Embodiment 5: The candidate box coordinates are converted from relative grid coordinates to absolute image coordinates by the following formula:

[0069] ;

[0070] in: and are the center coordinates of the X axis and the center coordinates of the Y axis of the absolute image coordinates respectively; and are the width and height of the candidate box coordinates adjusted to absolute image coordinates; Represents the Sigmoid function, which is used to limit the center coordinates to within the scope; is the scale adjustment coefficient, which is used to control the nonlinear change of the bounding box size.

[0071] Specifically, the core of Example 5 is to convert the candidate box generated by the detection model from relative grid coordinates to absolute image coordinates to ensure that the bounding box can be accurately positioned in the actual size of the target video frame. The YOLO model divides the input image into grids and predicts the bounding box of the target within each grid. However, the position and size of the bounding box predicted by the initial model are local coordinates relative to the grid, which means that they only describe the relative position of the target inside the grid. Therefore, in order to achieve accurate positioning on the entire frame image, these relative coordinates need to be transformed and mapped to the global coordinate system of the image. The key to this process is to convert the center coordinates of the candidate box by calculation. ,width and height Mapping from relative grid scale to absolute image size The center coordinates of the bounding box are given by the formula and Convert and Represents the horizontal and vertical index of the grid, respectively. is the number of grids, is the Sigmoid function, which is used to normalize the center coordinates to The Sigmoid function plays a role of smoothing constraint here, ensuring that the center coordinates of the candidate box will not exceed the grid boundary, but are strictly limited to the inside of the grid. Through such a transformation, the center position of the bounding box can be accurately mapped to the global coordinates of the image, ensuring the accuracy of the target's position information.

[0072] In terms of candidate box resizing, the width and height The conversion is done by using the exponential function This is because the size of the target may vary greatly in different scenarios, and the nonlinear scaling characteristics of the exponential function can flexibly adjust the size of the bounding box. Specifically, the converted width and height are respectively and Given, where is the scale adjustment factor, which is used to control the degree of size scaling. , the size of the bounding box can be flexibly adjusted, making the detection of fireworks targets more robust in different scenarios. This nonlinear resizing mechanism, combined with the growth characteristics of the exponential function, allows the size of the bounding box to adapt to the complex shapes and size changes of flames and smoke. Flames and smoke have irregular shapes and diverse diffusion ranges. This adjustment method can ensure that the size of the bounding box matches the actual size of the target. Converting the candidate box from relative grid coordinates to absolute image coordinates is not only to correspond the target detection box to the actual position of the original image, but also to ensure that the center and size of the bounding box have a clear physical meaning in the target frame. This is crucial for fireworks detection because flame and smoke targets usually show large scale changes and position instability. Mapping the coordinates to the global image coordinate system allows the position and size of the detection box to accurately reflect the position and size of the target in the image, further improving the accuracy and robustness of the detection.

[0073] Example 6: Use the following formula to perform non-maximum suppression on the adjusted candidate boxes to remove duplicate detection boxes:

[0074] ;

[0075] if ; then remove ;in: Candidate box With candidate box The intersection and union ratio of is the overlapping area of ​​the two candidate boxes; is the total area of ​​the two candidate boxes; and All are subscript indexes.

[0076] Specifically, during the target detection process, the model generates multiple bounding boxes in each grid to capture the characteristics of flames and smoke. However, this method of multi-bounding box prediction can easily cause the same target to be covered by multiple candidate boxes, especially when the fireworks target has a complex shape, a large area, or a lot of background noise, there may be multiple overlapping bounding boxes locating the same target. Therefore, in order to improve the accuracy and simplicity of the detection results, NMS filters and merges overlapping candidate boxes by calculating the intersection over union (IoU) between bounding boxes. The specific process of non-maximum suppression is: for each set of overlapping bounding boxes, the system selects the box with the highest confidence as the final retained bounding box, and deletes those whose overlap with it is higher than the set threshold. This process first sorts all candidate boxes from high to low according to their confidence, and then starts with the bounding box with the highest confidence and calculates the intersection and union ratio between it and the remaining candidate boxes. If the intersection and union ratio exceeds the threshold , indicating that the overlap between the two bounding boxes is too high, the model will delete the bounding box with lower confidence and only keep the one with the highest confidence. It is usually set between 0.5 and 0.7 to ensure that only those candidate boxes with obvious overlap are merged, while boxes with relatively independent boundaries can be retained to represent different targets. This process is repeated until all candidate boxes have been processed. Through non-maximum suppression, the number of candidate boxes can be significantly reduced, duplicate detections can be eliminated, and each detected fireworks target can be represented by a unique bounding box. This is very critical for fireworks detection, because flames and smoke may change rapidly in actual scenes, have various shapes, and have unclear boundaries with other objects. NMS helps reduce duplicate detections caused by complex targets and improves the clarity and accuracy of detection. It can also reduce the redundant information output by the model while ensuring detection accuracy, providing a more concise set of bounding boxes for subsequent dynamic analysis and abnormal behavior identification.

[0077] Embodiment 7: In step 2, the process of using the frame difference-based method to determine whether there is a moving object in the current frame of the target video, and calculating the intersection-over-union ratio of the moving object and the target detection frame set as the abnormality ratio specifically includes: and the previous frame , calculate the gradient difference ; Introduce a multi-threshold adaptive strategy to calculate the local threshold according to the gradient characteristics of different regions and the nonlinear enhancement coefficient , for gradient difference Perform binarization processing to obtain the binarization result ; Binarization result Connected domain detection is performed to identify continuous motion areas. The intersection-and-union ratio of the identified motion area and the set of target detection frames is calculated. If the intersection-and-union ratio of the motion area and the set of target detection frames exceeds the set threshold, the abnormality ratio is calculated and the current frame is marked.

[0078] Specifically, by and the previous frame Perform gradient difference calculation to obtain gradient difference ,in Represents the pixel coordinates in the image. Gradient difference is a key technology used to highlight motion areas in inter-frame difference detection. Because the dynamic changes of flames and smoke in the image are more obvious, there will be significant gradient changes in the area containing the fireworks target between the current frame and the previous frame. By calculating the gradient difference between two adjacent frames, the system can quickly locate the area where motion changes occur in the frame. Gradient difference calculation is essentially an analysis of the intensity changes at each pixel position to find those pixels that change greatly over time. However, since video frames may contain multiple types of dynamic changes, such as light changes, noise, background movement, etc., relying solely on a single threshold for differential detection may result in a large number of false alarms and missed detections. In order to solve this problem, the present invention introduces a multi-threshold adaptive strategy. The core idea of ​​the multi-threshold adaptive strategy is to calculate the local threshold based on the gradient characteristics of different regions in the image. , and the nonlinear enhancement coefficient This adaptive threshold adjustment method can more accurately adapt to the dynamic characteristics of different areas in the image. For example, in areas where the light intensity changes significantly, a higher threshold can be used to avoid false positives; while in areas where the target features are more prominent, a lower threshold can be used to ensure that the target motion is captured. This strategy ensures that different areas can be treated differently in complex scenes, improving the accuracy and robustness of motion detection.

[0079] Next, we take the gradient difference Binarization is to convert the difference result into an image containing only "0" or "1". 1 indicates that there is a significant change in the pixel between the current frame and the previous frame, that is, the motion area, and 0 indicates no change or static area. Binarization is done by thresholding. and the nonlinear enhancement coefficient The latter is used to amplify the pixel changes that are important in motion detection, so as to better highlight the moving objects. Through this processing step, the system can extract the obvious dynamic areas in the target video, which are likely to contain abnormal targets such as flames and smoke. After that, it is subjected to connected domain detection. Connected domain detection is the process of identifying continuous motion regions in the binarization result. It determines which pixels belong to the same moving object by analyzing the connectivity between adjacent pixels. Fire and smoke targets usually have coherent regional characteristics, so connected domain detection can separate these targets from the binary image. By classifying continuous motion pixels into the same connected domain, moving objects in the video can be better extracted to avoid false detections caused by noise or background changes. After identifying the motion region, it is necessary to compare these motion regions with the target detection frame. Specifically, the intersection over union (IoU) between the motion region of each connected domain and the set of target detection frames is calculated to determine whether the motion region matches the detected fireworks target. The IoU is defined as the ratio of the intersection area to the union area of ​​two regions, which reflects the degree of overlap between the motion region and the detection frame. If the IoU exceeds the set threshold, it means that the detected motion region is highly overlapped with the fireworks target, that is, it can be considered that the motion region may be related to the fireworks anomaly. Once the IoU of the motion region and the target detection frame exceeds the set threshold, the system calculates the anomaly ratio. The abnormality ratio is an indicator that measures the degree of overlap between the motion area and the target detection frame in the current frame, reflecting the proportion of abnormal motion targets that may exist in the frame. If the abnormality ratio reaches or exceeds the preset threshold, the current frame will be marked as a frame that may contain fireworks abnormalities, thereby providing a reference for subsequent alarms and further processing.

[0080] Example 8: Use the following formula to calculate the current frame and the previous frame , calculated in different directions and different scales Gradient difference on :

[0081] ;

[0082] in, is the gradient difference, which means in the coordinate The difference value at ; For the direction The Gaussian filter on , with a scale parameter of ; Indicates in the direction Gradient calculation on ; Represents the convolution operation; is the nonlinear enhancement coefficient, which controls the degree of enhancement of the difference value; the binarization result is obtained through the following formula: :

[0083] ;

[0084] in, , is the center of the grid The neighborhood of the center The average difference of is the local dynamic smoothing coefficient, ; is the set spatiotemporal smoothing parameter; is the nonlinear mapping coefficient; is the X-axis coordinate of the center of the grid; The Y-coordinate of the center of the grid.

[0085] Specifically, calculate the current frame and the previous frame Gradient difference of , which is the basis of the entire motion detection. In the calculation process, the Gaussian filter and gradient calculation Gaussian filter is a smoothing operator that and scale Convolving an image at a certain scale can highlight the features in the image at that direction and scale. Gaussian filters of different scales can capture changes of different sizes and frequencies in the image, while the choice of direction helps detect edges or motion in a specific direction. and multi-directional By calculating the gradient, we can obtain dynamic information about the target in all directions and scales, ensuring that the complex morphological changes of the fireworks target are fully captured. It is represented by the absolute value of the difference in the directional gradient convolution of the two frames. This calculation method retains the motion change characteristics between the two frames and uses the nonlinear enhancement coefficient These changes are further amplified, making the significant motion features more prominent in the gradient difference. After calculating the gradient difference, how to effectively extract the moving target becomes the key. Since fireworks targets usually show non-stationary and rapidly changing characteristics, their gradient difference values ​​in local areas will be significantly higher than those in the background area. In order to accurately extract these moving areas, a binarization strategy is adopted to convert the gradient difference into Convert to binary image , in order to identify possible moving objects. Traditional binarization methods usually rely on global or local fixed thresholds, but in complex fireworks scenes, this simple threshold processing is easily affected by background interference and noise. The present invention uses an adaptive threshold strategy to dynamically adjust the threshold according to local image characteristics, so that the system can better adapt to the dynamic characteristics in different scenes, thereby improving the accuracy of detection.

[0086] The setting of the adaptive threshold depends on the average gradient difference value of the grid center neighborhood , which means the grid center The average value of the gradient change in the local area centered on . By averaging the gradient difference in this area, the noise can be better smoothed out and the main motion information can be retained. At the same time, the local dynamic smoothing coefficient is introduced in the adjustment item , used to enhance image edge and motion features. This coefficient is determined by the second-order derivative of the gradient difference, specifically including , And the second-order partial derivative in the mixing direction. This processing method ensures a stronger response in areas where the image gradient changes significantly (such as the edge of the flame), enhancing the sensitivity to moving targets. Local dynamic smoothing coefficient By calculating the second-order derivative of the gradient difference, the strong edge features in the moving area are amplified, making these features easier to identify and extract during the binarization process. In addition, in order to ensure that the binarization result can better meet the dynamic characteristics of the actual target, the present invention introduces two control parameters: and . is the spatiotemporal smoothing parameter, which controls the smoothness of the gradient difference in space. , the system can smoothly analyze the changes of gradient differences in the spatial neighborhood, thereby ensuring the continuity of the detection results in time and space and avoiding false alarms caused by noise or instantaneous motion. On the other hand, is the nonlinear mapping coefficient, which adjusts the relationship between the gradient difference value and the threshold. Nonlinear mapping can amplify those motion areas with obvious but not significant differences, making the detection more accurate. In the binarization process, a conditional function is set to judge Is it greater than the adaptive threshold? If this condition is met, If set to 1, it means there is significant motion at the current position; otherwise, it is set to 0, indicating no significant motion. This judgment condition not only takes into account the size of the gradient difference, but also introduces an exponential decay function to ensure that the influence of pixels far from the center position on the threshold gradually weakens. This design can further enhance the detection of local significant change areas and suppress spatially dispersed noise.

[0087] Example 9: The process of calculating the abnormality ratio specifically includes: assuming For the detected connected domain set, calculate the weighted area of ​​each connected domain :

[0088] ;

[0089] in, Connected domain The weighted area of ​​? is a nonlinear weight, determined according to the image gradient change; is the number of connected domains; let the target detection box set , then calculate the intersection and union ratio of the identified motion area and the target detection frame set :

[0090] ;

[0091] in, is the number of candidate boxes in the target detection box set; Represents the first Candidate boxes; is the first Connected domains; calculate the anomaly ratio using the following formula: :

[0092] .

[0093] Specifically, in actual scenes, flames and smoke are highly irregular and dynamically changing targets, and they usually have complex interactions with the surrounding background. Therefore, it is difficult to accurately capture these features only through simple target detection or inter-frame difference. The design of the present invention aims to refine the correlation between the motion area and the detection frame by calculating the weighted area and weighted intersection-over-union ratio of the connected domain, so as to obtain the abnormal proportion and ensure more accurate and comprehensive detection of fireworks targets. First, it is necessary to determine the set of all motion areas in the video frame, that is, the connected domain set Each connected domain It is an independent moving target, which is composed of pixels connected to each other after binary processing. Since the movement of fireworks targets is complex and changeable, these connected domains are usually the areas where movement occurs in the frame, which may be flames burning, smoke spreading, or other possible dynamic changes. The number of these connected domains may be different in each frame, depending on the video content and target detection.

[0094] In order to accurately evaluate the importance of each connected domain in the image, the present invention introduces a weighted area , which is used to measure the area size of the connected domain and combine its features. The weighted area here is not a simple accumulation of the number of pixels, but a weighted processing of the gradient change of each pixel. The core idea is that the edge of the fireworks target is usually accompanied by a high gradient change. If the pixel area is used directly as an indicator, it may be affected by edge noise and false detection. Therefore, the weight coefficient is introduced To reduce the impact of high gradient areas and ensure that the calculated area can truly reflect the characteristics of the motion area. It is defined based on the gradient change of pixels, and the absolute value of the gradient is introduced in the specific formula. Pixels with large gradients usually appear at the edges or where the texture changes rapidly. In fireworks detection, over-emphasizing these areas may introduce unnecessary noise. Therefore, the weights of these high-gradient pixels are reduced through the weight coefficient, making their contribution to the overall area smaller, ensuring the weight area of ​​the connected domain After determining the weighted area of ​​each connected domain, the next step is to match these motion regions with the target detection boxes pre-detected by the YOLO model. Contains possible fireworks targets detected in the current frame, each detection box is a candidate region. In order to evaluate the motion region With detection box The correlation between them needs to calculate the spatial overlap between them. The traditional intersection over union (IoU) is usually used to measure the overlap between two regions. However, due to the complex edge changes and irregular shapes of fireworks targets in images, the present invention improves the calculation method of IoU and considers the influence of the weight coefficient on the overlap area between the connected domain and the detection frame.

[0095] Specifically, when calculating the weighted IoU in the present invention, the intersection and union of the connected domain and the detection frame are adjusted by the weight coefficient. In this way, in the intersection part, the pixels with larger weights contribute more to the IoU, and in the union part, the actual importance of spatial coverage is also measured according to the weight. This design ensures that the IoU can more accurately reflect the matching degree between the motion area and the detection frame, and reduces the interference of noise and error on the matching results. Finally, in order to obtain the abnormal proportion of the current frame , it is necessary to combine the weighted IoU between all connected domains and the detection box. It represents the overall matching degree between the motion region and the detection frame, and is a key indicator to measure whether there is an abnormality in the current frame. When calculating the abnormality ratio, the weighted area of ​​each connected domain and its weighted IoU with the target detection frame are taken into account, so that those motion regions with larger areas and highly matched with the detection frame contribute more to the abnormality ratio. In this way, the abnormality ratio not only reflects the overlap degree between the motion region and the detection frame, but also combines the concept of weighted area, so that important areas with larger areas have a greater impact on the detection results. This weighted processing method is very effective in complex scenes because it can suppress errors that may be caused by background noise, false detection areas, and edge effects, while highlighting those motion regions that are truly related to fireworks targets. The calculation of the abnormality ratio is the core of the entire fireworks detection process because it directly determines the system's judgment result on the current frame. When the abnormality ratio exceeds the preset threshold, the system will mark the frame as a possible abnormality, thereby providing a basis for subsequent alarms and responses. If the abnormality ratio is lower than the threshold, the frame is considered normal, and the possible motion area is not related to fireworks. This strategy based on anomaly ratio enables the system to quickly and accurately identify fireworks targets in practical applications, improving the sensitivity and robustness of detection.

[0096] Although the specific embodiments of the present invention are described above, it should be understood by those skilled in the art that these specific embodiments are only illustrative, and those skilled in the art may omit, replace, and change the details of the above methods and systems in various ways without departing from the principles and essence of the present invention. For example, merging the above method steps so as to perform substantially the same functions in substantially the same manner to achieve substantially the same results is within the scope of the present invention. Therefore, the scope of the present invention is limited only by the appended claims.

Claims

1. A method for detecting and analyzing fireworks based on the YOLO algorithm and dynamic analysis, characterized in that: The method comprises: Step 1: Use the trained fireworks detection model to detect fireworks candidate targets in the target video and record their coordinate positions as the target detection box set; Step 2: Use the frame difference-based method to determine whether there is a moving object in the current frame of the target video, and calculate the intersection-and-union ratio of the moving object and the target detection frame set as the abnormality ratio. If the abnormality ratio exceeds the set abnormality discrimination threshold, start timing and mark this frame as abnormal, otherwise mark this frame as normal, and repeat the cycle until the predetermined time is met; Step 3: If the abnormality ratio does not exceed the set abnormality discrimination threshold, the first recorded frame will be released, and the newly added frame information will be recorded. If there is no abnormal frame record for the set number of consecutive frames, the recorded frame information will be cleared; if the abnormality ratio meets the requirement, the recorded abnormal frame information will be judged for linear motion. If it is judged to be non-linear motion, an alarm message will be output. If it is judged to be linear motion, the record will be cleared and no alarm message will be output; Step 1 specifically includes: for each frame of the target video Perform preprocessing, adjust the image size to the target value and normalize it; Represents the index of the time frame, which indicates the first The image of the frame is in the range of ,in is the total number of frames of the target video; the pre-trained fireworks detection model is used to infer each frame of the pre-processed image to obtain the position and confidence of the detection frame; the fireworks detection model uses a neural network to divide each frame into Grids, each grid predicts multiple bounding boxes and corresponding confidence and category probabilities; the output of the fireworks detection model is represented as a three-dimensional tensor: ;in, is the output tensor containing the prediction information for each grid; is the number of grid divisions; The number of bounding boxes predicted for each grid; Indicates that each bounding box contains 5 parameters: center coordinates ,width ,high And confidence ; Indicates the X-axis coordinate; Indicates the Y-axis coordinate; is the number of categories, indicating the probability of fireworks categories; the bounding boxes output by the model are filtered and the ones with confidence levels higher than the threshold are retained The bounding box of the image is used as the candidate box; the coordinates of the candidate box are adjusted from the relative grid coordinates to the absolute image coordinates; non-maximum suppression is performed on the adjusted candidate box to remove duplicate detection boxes; for each adjusted candidate box, the intersection and union ratio with other boxes is calculated, and if the intersection and union ratio is higher than the set threshold , then the candidate box with the highest confidence is retained to form the target detection box set; represents the field of real numbers; In step 2, the method based on frame difference is used to determine whether there is a moving object in the current frame of the target video, and the intersection and union ratio of the moving object and the target detection frame set is calculated as the abnormality ratio. The specific process includes: and the previous frame , calculate the gradient difference ; Introduce a multi-threshold adaptive strategy to calculate the local threshold according to the gradient characteristics of different regions and the nonlinear enhancement coefficient , for gradient difference Perform binarization processing to obtain the binarization result ; Binarization result Perform connected domain detection to identify continuous motion areas; calculate the intersection-and-union ratio of the identified motion area and the target detection frame set. If the intersection-and-union ratio of the motion area and the target detection frame set exceeds the set threshold, calculate the abnormality ratio and mark the current frame; The specific process of calculating the abnormality ratio includes: For the detected connected domain set, calculate the weighted area of ​​each connected domain : ； in, Connected domain The weighted area of ​​? is a nonlinear weight, determined according to the image gradient change; is the number of connected domains; let the target detection box set , then calculate the intersection and union ratio of the identified motion area and the target detection frame set : ； in, is the number of candidate boxes in the target detection box set; Represents the first Candidate boxes; is the first Connected domains; calculate the anomaly ratio using the following formula: : ； In step 3, the linear motion judgment is performed on the recorded abnormal frame information through the following process: The target is detected in the frame, and its position coordinates are , then the target's motion trajectory is expressed as a time series ,in is the frame number, and are the horizontal and vertical coordinates of the target in the frame respectively; in the continuous frames, the motion trajectory of the target is obtained by tracking the position change of the target detection frame; the time series Fitting into a straight line to evaluate whether the target's motion trajectory is approximately a straight line motion; performing linear fitting by the least squares method to obtain the parametric equation of the fitting line; the goal of the fitting process is to find a straight line so that the distance between the target's motion trajectory points and this straight line is minimized; after the linear fitting is completed, the residual from the trajectory points to the fitting line is calculated, and the sum of the squares of the residuals is usually used to measure the quality of the fit; in order to determine whether the target's motion trajectory is a linear motion, the present invention introduces a linear discrimination criterion; if the target's motion trajectory presents a regular straight line motion, then the residual value should be less than the set residual threshold, and the trajectory points are evenly distributed near the fitting line; therefore, by setting the residual threshold To determine whether the target's motion trajectory is linear: if , it is judged as linear motion, otherwise, it is judged as nonlinear motion.

2. The method for smoke detection and analysis based on YOLO algorithm and dynamic analysis as claimed in claim 1, characterized in that: The fireworks detection model is YOLOv8s.

3. The method for smoke detection and analysis based on YOLO algorithm and dynamic analysis as claimed in claim 2, characterized in that: Assume that the size of each frame of the target video is , resize it to the target size , and then normalize it; let the horizontal direction , vertically The grid The confidence of the predicted bounding box is expressed as ; The bounding box output by the model is filtered using the following formula: ； Among them, in the mesh division of the smoke and fire detection model, and Represents the position index of the grid. The fireworks detection model divides each frame into The grid, and Respectively represent the index position of the grid in the vertical and horizontal directions; the ranges are and ; Represents the index of the predicted bounding box in each grid; the fire detection model predicts multiple bounding boxes for each grid to capture different target candidate boxes; The range is ,in is the number of bounding boxes predicted for each grid; is the set of candidate boxes after filtering; The horizontal direction after filtering , vertically The first The coordinates and sizes of the candidate boxes corresponding to the predicted bounding boxes, is the corresponding X-axis coordinate, is the corresponding Y-axis coordinate, is the corresponding width; The corresponding height.

4. The method for smoke detection and analysis based on YOLO algorithm and dynamic analysis as claimed in claim 3, characterized in that: The candidate box coordinates are converted from relative grid coordinates to absolute image coordinates using the following formula: ； in: and are the center coordinates of the X axis and the center coordinates of the Y axis of the absolute image coordinates respectively; and are the width and height of the candidate box coordinates adjusted to absolute image coordinates; Represents the Sigmoid function, which is used to limit the center coordinates to within the scope; is the scale adjustment coefficient, which is used to control the nonlinear change of the bounding box size.

5. The method for smoke detection and analysis based on YOLO algorithm and dynamic analysis as claimed in claim 4, characterized in that: Use the following formula to perform non-maximum suppression on the adjusted candidate boxes to remove duplicate detection boxes: ； if ; then remove ;in: Candidate box With candidate box The intersection and union ratio of is the overlapping area of ​​the two candidate boxes; is the total area of ​​the two candidate boxes; and All are subscript indexes.

6. The method for smoke detection and analysis based on YOLO algorithm and dynamic analysis as claimed in claim 5, characterized in that: Use the following formula for the current frame and the previous frame , calculated in different directions and different scales Gradient difference on : ; in, is the gradient difference, which means in coordinates The difference value at ; For the direction The Gaussian filter on , with a scale parameter of ; Indicates in the direction Gradient calculation on ; Represents the convolution operation; is the nonlinear enhancement coefficient, which controls the degree of enhancement of the difference value; the binarization result is obtained through the following formula: : ； in, , is the center of the grid The neighborhood of the center The average difference of is the local dynamic smoothing coefficient, ; is the set spatiotemporal smoothing parameter; is the nonlinear mapping coefficient; is the X-axis coordinate of the center of the grid; The Y-coordinate of the center of the grid.

Images