Flame detection method and system based on LSTM multi-frame sequence
Through the combination of LSTM multi-frame sequence neural network and ByteTrack algorithm, the flame trajectory is generated and tracked, and the existing flame detection methods are solved, and high-precision flame detection is achieved in complex backgrounds.
Patent Information
- Application Number
- CN202410993483.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-15
AI Technical Summary
The existing flame detection methods are slow to respond and are susceptible to the environment, resulting in low detection accuracy, especially in complex backgrounds, frequent misjudgment, difficult to process high-definition video streams in real time, and difficult to capture dynamic flame characteristics.
Using the flame detection method based on LSTM multi-frame sequence, the ByteTrack algorithm is used to generate flame trajectories, and through the pre-constructed flame detection model, combined with the LSTM multi-frame sequence neural network, the training model takes the historical flame trajectory as input and the labeled data as output, and optimizes the Network weight, realize flame detection.
This method can quickly respond and continuously track potential flame regions, reduce false alarms, improve the accuracy of flame detection in complex backgrounds, and detect real flame dynamics from continuous frames.
Smart Images

Figure CN118968136A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of flame detection, and in particular to a flame detection method and system based on LSTM multi-frame sequence. Background Art
[0002] Existing flame detection methods mainly include: smoke detection, temperature detection, visual smoke detection, etc. Among them, smoke detection and temperature detection have shortcomings such as slow response and susceptibility to environmental influences. Visual smoke detection has the following problems: (1) The appearance of the flame will be affected by environmental factors (such as lighting conditions and background complexity). For example, strong sunlight or reflections may interfere with the flame detection system and cause misjudgment; it is a challenge to develop a model that can accurately identify flames under various lighting conditions and complex backgrounds. (2) Real-time requirements: When a fire occurs, rapid response is essential. Therefore, the flame detection system needs to be able to process high-definition video streams in real time, which places high demands on computing resources and algorithm optimization. (3) Distinguishing between flames and other objects: In certain circumstances, such as the setting sun, reflected light sources or tungsten light from bulbs, they may be visually similar to flames, making it difficult to accurately detect flames. (4) Capturing dynamic flame features: The shape and intensity of the flame will change rapidly over time, which requires the detection algorithm to capture and identify dynamically changing features. Summary of the invention
[0003] The present invention provides a flame detection method based on LSTM multi-frame sequence, which is used to solve the problems that the existing flame detection method has slow response, is easily affected by the environment, and has low detection accuracy due to misjudgment of detection, including:
[0004] Obtaining image data information of the area to be identified;
[0005] When there is a potential flame area in the image data information, generating a flame track using a ByteTrack algorithm based on the image data information corresponding to the potential flame area;
[0006] Based on the flame trajectory, using a pre-built flame detection model, a flame detection result is obtained to determine whether the flame trajectory is a real flame phenomenon;
[0007] The flame detection model is obtained by training the LSTM multi-frame sequence neural network using historical flame trajectories as input of training data and labeled data corresponding to the historical flame trajectories as output of training data.
[0008] Optionally, the potential flame area includes the following judgment process:
[0009] According to a preset flame image database, using a structural similarity index method, calculating the similarity between the data in the flame image database and the image data information;
[0010] When the similarity is greater than a preset similarity threshold, it is determined that there is a potential flame area in the graphic data information.
[0011] Optionally, generating a flame track by using a ByteTrack algorithm based on the image data information corresponding to the potential flame area includes:
[0012] Perform image preprocessing according to the image data information corresponding to the potential flame area to obtain processed image data;
[0013] Perform feature extraction based on the processed image data to obtain corresponding image feature data;
[0014] Automatically annotating each flame object in the image feature data to obtain a bounding box and a confidence score of each flame object;
[0015] Generate a flame track using a ByteTrack algorithm based on the bounding box of each flame object whose confidence score is greater than a set confidence threshold;
[0016] Wherein, the image preprocessing includes: data denoising, data enhancement and color space conversion;
[0017] The image feature data includes: color feature, texture feature, shape feature and temperature feature.
[0018] Optionally, the generating of the flame track based on the bounding box of each flame object having a confidence score greater than a set confidence threshold using a ByteTrack algorithm comprises:
[0019] According to the bounding boxes of each flame object whose confidence score is greater than a set confidence threshold, the overlap between the bounding boxes is calculated using the ByteTrack algorithm to perform bounding box matching;
[0020] According to the coordinate information of the successfully matched bounding box in each frame, the center points or key points of the successfully matched bounding box are connected to form a continuous trajectory line;
[0021] The continuous trajectory lines are superimposed and displayed to generate a flame trajectory.
[0022] Optionally, the flame detection model includes the following construction process:
[0023] Acquire a continuous historical video sequence, which includes actual flame trajectories and false flame trajectories of the flame;
[0024] Extract frames from the historical video sequence, and for each frame or selected key frame, accurately mark the flame area using automated tools, including but not limited to bounding box positioning, pixel-level segmentation or key point marking, to generate corresponding annotation data;
[0025] The actual flame trajectories and false flame trajectories corresponding to the labeled data of adjacent frames are organized into multi-frame sequences in chronological order, and the flame position information of multiple consecutive frames in each sequence is used as the input of training data;
[0026] Outputting the labeled data corresponding to the flame position information as training data;
[0027] Based on the input and output of the training data, the LSTM multi-frame sequence neural network is trained, and a cross entropy loss function is used to optimize the network weights of the LSTM multi-frame sequence neural network to obtain a flame detection model.
[0028] Optionally, after obtaining the flame detection result of whether the flame trajectory is a real flame phenomenon, the method further includes:
[0029] When the flame trajectory is a real flame phenomenon, obtaining the regional coordinate information of the image data information corresponding to the flame trajectory, and judging the fire level according to the flame trajectory;
[0030] If the fire level is a minor fire, the local fire extinguishing device is automatically triggered;
[0031] If the fire level is medium, an alarm message will be automatically sent to the backend;
[0032] If the fire level is a serious fire, an alarm will be automatically sounded.
[0033] Based on the same inventive concept, the present invention also provides a flame detection system based on LSTM multi-frame sequence, comprising:
[0034] Data acquisition module: used to obtain image data information of the area to be identified;
[0035] Trajectory generation module: used for generating flame trajectory by using ByteTrack algorithm based on the image data information corresponding to the potential flame area when there is a potential flame area in the image data information;
[0036] Flame detection module: used to obtain a flame detection result of whether the flame trajectory is a real flame phenomenon based on the flame trajectory using a pre-built flame detection model;
[0037] The flame detection model in the flame detection module is obtained by training the LSTM multi-frame sequence neural network using historical flame trajectories as input of training data and labeled data corresponding to the historical flame trajectories as output of training data.
[0038] Optionally, the potential flame area in the trajectory generation module includes the following judgment process:
[0039] According to a preset flame image database, using a structural similarity index method, calculating the similarity between the data in the flame image database and the image data information;
[0040] When the similarity is greater than a preset similarity threshold, it is determined that there is a potential flame area in the graphic data information.
[0041] Optionally, the trajectory generation module generates a flame trajectory using a ByteTrack algorithm based on the image data information corresponding to the potential flame area, including:
[0042] Perform image preprocessing according to the image data information corresponding to the potential flame area to obtain processed image data;
[0043] Perform feature extraction based on the processed image data to obtain corresponding image feature data;
[0044] Automatically annotating each flame object in the image feature data to obtain a bounding box and a confidence score of each flame object;
[0045] Generate a flame track using a ByteTrack algorithm based on the bounding box of each flame object whose confidence score is greater than a set confidence threshold;
[0046] Wherein, the image preprocessing includes: data denoising, data enhancement and color space conversion;
[0047] The image feature data includes: color feature, texture feature, shape feature and temperature feature.
[0048] Optionally, the trajectory generation module generates a flame trajectory using a ByteTrack algorithm based on the bounding box of each flame object whose confidence score is greater than a set confidence threshold, including:
[0049] According to the bounding boxes of each flame object whose confidence score is greater than a set confidence threshold, the overlap between the bounding boxes is calculated using the ByteTrack algorithm to perform bounding box matching;
[0050] According to the coordinate information of the successfully matched bounding box in each frame, the center points or key points of the successfully matched bounding box are connected to form a continuous trajectory line;
[0051] The continuous trajectory lines are superimposed and displayed to generate a flame trajectory.
[0052] Optionally, the flame detection model in the flame detection module includes the following construction process:
[0053] Acquire a continuous historical video sequence, which includes actual flame trajectories and false flame trajectories of the flame;
[0054] Extract frames from the historical video sequence, and for each frame or selected key frame, accurately mark the flame area using automated tools, including but not limited to bounding box positioning, pixel-level segmentation or key point marking, to generate corresponding annotation data;
[0055] The actual flame trajectories and false flame trajectories corresponding to the labeled data of adjacent frames are organized into multi-frame sequences in chronological order, and the flame position information of multiple consecutive frames in each sequence is used as the input of training data;
[0056] Outputting the labeled data corresponding to the flame position information as training data;
[0057] Based on the input and output of the training data, the LSTM multi-frame sequence neural network is trained, and a cross entropy loss function is used to optimize the network weights of the LSTM multi-frame sequence neural network to obtain a flame detection model.
[0058] Optionally, after the flame detection module obtains the flame detection result of whether the flame trajectory is a real flame phenomenon, it also includes:
[0059] When the flame trajectory is a real flame phenomenon, obtaining the regional coordinate information of the image data information corresponding to the flame trajectory, and judging the fire level according to the flame trajectory;
[0060] If the fire level is a minor fire, the local fire extinguishing device is automatically triggered;
[0061] If the fire level is medium, an alarm message will be automatically sent to the backend;
[0062] If the fire level is a serious fire, an alarm will be automatically sounded.
[0063] In yet another aspect, the present invention further provides a computer device, comprising: one or more processors;
[0064] A memory for storing one or more programs;
[0065] When the one or more programs are executed by the one or more processors, a flame detection method based on LSTM multi-frame sequence as described above is implemented.
[0066] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed, the flame detection method based on the LSTM multi-frame sequence as described above is implemented.
[0067] Compared with the closest prior art, the present invention has the following beneficial effects:
[0068] The present invention provides a flame detection method and system based on LSTM multi-frame sequence, comprising: acquiring image data information of a to-be-identified area; when there is a potential flame area in the image data information, generating a flame trajectory by using a ByteTrack algorithm based on the image data information corresponding to the potential flame area; obtaining a flame detection result of whether the flame trajectory is a real flame phenomenon by using a pre-constructed flame detection model based on the flame trajectory; wherein the flame detection model is obtained by training an LSTM multi-frame sequence neural network with historical flame trajectories as input of training data and with labeled data corresponding to the historical flame trajectories as output of training data; the present invention combines the ByteTrack algorithm with a flame detection model constructed based on the LSTM multi-frame sequence neural network, can quickly respond to and continuously track potential flame areas, can maintain tracking accuracy even in dynamic scenes, can detect real flame dynamics from a series of continuous frames, can reduce false alarms even in complex backgrounds, and is conducive to improving the accuracy of flame detection.
[0069] Other features of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.
[0070] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:
[0072] Figure 1 A schematic diagram of a flame detection method based on LSTM multi-frame sequence provided by the present invention;
[0073] Figure 2A schematic diagram of the training process of a flame detection model in a flame detection method based on LSTM multi-frame sequence provided by the present invention;
[0074] Figure 3 A schematic diagram of the overall architecture of a flame detection method based on LSTM multi-frame sequence provided by the present invention;
[0075] Figure 4 A schematic diagram of the structural composition of a flame detection system based on LSTM multi-frame sequence provided by the present invention. DETAILED DESCRIPTION
[0076] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.
[0077] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "Multiple" means two or more, unless otherwise clearly and specifically defined. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0078] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
[0079] Embodiment 1:
[0080] The present invention provides a flame detection method based on LSTM multi-frame sequence, the flow chart is as follows Figure 1 As shown, including:
[0081] Step 1: Obtain image data information of the area to be identified;
[0082] Step 2: when there is a potential flame area in the image data information, a flame track is generated using a ByteTrack algorithm based on the image data information corresponding to the potential flame area;
[0083] Step 3: Based on the flame trajectory, using a pre-built flame detection model, a flame detection result is obtained to determine whether the flame trajectory is a real flame phenomenon;
[0084] The flame detection model is obtained by training the LSTM multi-frame sequence neural network using historical flame trajectories as input of training data and labeled data corresponding to the historical flame trajectories as output of training data.
[0085] The image data information in the above step 1 can be collected in real time through the video acquisition and processing system to the image data in the area to be identified, and the data can be sent back to the server; it is installed in the flame to be identified area to ensure that there is complete image coverage in the monitored area, and each installation location has a fixed location code to ensure that the corresponding location can be found according to the video acquisition system.
[0086] In one implementation, the potential flame area in step 2 above may include the following judgment process:
[0087] According to a preset flame image database, using a structural similarity index method, calculating the similarity between the data in the flame image database and the image data information;
[0088] When the similarity is greater than a preset similarity threshold, it is determined that there is a potential flame area in the graphic data information.
[0089] In one implementation, the process of generating the flame track using the ByteTrack algorithm based on the image data information corresponding to the potential flame area may include:
[0090] Perform image preprocessing according to the image data information corresponding to the potential flame area to obtain processed image data;
[0091] Perform feature extraction based on the processed image data to obtain corresponding image feature data;
[0092] Automatically annotating each flame object in the image feature data to obtain a bounding box and a confidence score of each flame object;
[0093] Generate a flame track using a ByteTrack algorithm based on the bounding box of each flame object whose confidence score is greater than a set confidence threshold;
[0094] Wherein, the image preprocessing includes: data denoising, data enhancement and color space conversion;
[0095] The image feature data includes: color feature, texture feature, shape feature and temperature feature.
[0096] In this implementation, the process of generating flame tracks based on the bounding box of each flame object having a confidence score greater than a set confidence threshold using the ByteTrack algorithm may include:
[0097] According to the bounding boxes of each flame object whose confidence score is greater than a set confidence threshold, the overlap between the bounding boxes is calculated using the ByteTrack algorithm to perform bounding box matching;
[0098] According to the coordinate information of the successfully matched bounding box in each frame, the center points or key points of the successfully matched bounding box are connected to form a continuous trajectory line;
[0099] The continuous trajectory lines are superimposed and displayed to generate a flame trajectory;
[0100] In this implementation, multiple frames of potential flame regions are tracked using ByteTrack and other tracking algorithms to form several flame tracks. ByteTrack divides all suspected flame region detection scores of the image into high-score and low-score parts according to a threshold value, such as 0.5. The high-score part usually corresponds to a more certain flame region, and the low-score part may contain more false detections and inaccurately detected flame regions. First, the initial matching is performed, and the high-score region is matched and tracked with the existing tracking track of the previous image. This step is usually tracked using the sort algorithm. The unmatched flame region is matched with other low-score detection results. The advantage of this is that all detection results are used as much as possible for matching, which improves the accuracy and robustness of tracking. For detection results that fail to match, the ByteTrack algorithm will retain these areas for a period of time (for example, 30 frames) and wait for these areas to appear again before continuing to match. For detection frames that do not match the tracking track but have high scores, the ByteTrack algorithm will create a new tracking track. Through the ByteTrack tracking algorithm, the flame area in the image is synthesized into an action track.
[0101] In one implementation, the flame detection model in step 3 above includes the following construction process: Figure 2 As shown:
[0102] Acquire a continuous historical video sequence, which includes actual flame trajectories and false flame trajectories of the flame;
[0103] Extract frames from the historical video sequence, and for each frame or selected key frame, accurately mark the flame area using automated tools, including but not limited to bounding box positioning, pixel-level segmentation or key point marking, to generate corresponding annotation data;
[0104] The actual flame trajectories and false flame trajectories corresponding to the labeled data of adjacent frames are organized into multi-frame sequences in chronological order, and the flame position information of multiple consecutive frames in each sequence is used as the input of training data;
[0105] Outputting the labeled data corresponding to the flame position information as training data;
[0106] Based on the input and output of the training data, the LSTM multi-frame sequence neural network is trained, and the network weights of the LSTM multi-frame sequence neural network are optimized by using a cross entropy loss function to obtain a flame detection model;
[0107] In this implementation, the flame detection model takes the collected video or image data as input, and uses this flame detection model to detect whether there is a flame in the collected data as output. The images in the area of the trajectory are used with the LSTM multi-frame sequence network to determine whether a flame fire is emitted in the potential flame area, and finally determine whether there is a flame fire. For example, the data processing process includes: scaling all the images in the trajectory to a uniform size of 128*128, and then arranging them to form a group of images with a resolution of 128*128, inputting this group of images into the LSTM multi-frame sequence network, outputting the classification result, and using a threshold value such as 0.5 to judge, if it is greater than 0.5, it is a flame, and if it is less than 0.5, it is not a flame.
[0108] In this implementation, in order to ensure data diversity, historical video sequences can collect data from the following areas: residential areas include home kitchens, living rooms, and garages; industrial and manufacturing areas include factories, processing plants, and warehouses; commercial places include restaurants, Internet cafes, shopping centers, and office buildings; transportation vehicles include cars, ships, railways, and airplanes; forests and wild areas include campsites, forests, and grasslands; educational institutions include educational places such as schools and universities; public facilities include power stations, gas stations, garbage disposal plants, and charging stations, etc., where fires are prone to occur. The specific data of fires and non-fires sent, the flame area is annotated with data using automated tools, and the data without fire is annotated as normal data; in view of the large number of scenes, large amount of data, and slow annotation speed, the present invention can select the following technical directions for improvement to address this problem:
[0109] a Use specialized image annotation platforms (such as Labelbox, Supervisely, CVAT, etc.), which integrate pre-annotation algorithms, interactive editing, batch processing and other functions to significantly improve annotation efficiency.
[0110] b. Sample data from multiple sensors such as thermal infrared and smoke detectors for annotation, and perform consistency checks on different modal data of the same scene to improve the efficiency and speed of annotation and reduce mislabeling and omissions.
[0111] c. Based on the fire risk level or actual needs, data in high-risk areas (such as chemical plants, forests, schools, etc.) should be labeled first to ensure the labeling coverage of key scenes and the balanced ratio of fire and normal samples in the labeled data to avoid category skew during model training. In addition, the above flame detection model can also be trained and optimized using yolov5. Using the active learning strategy, a part of the data is first labeled in small batches for training, allowing the model to make preliminary predictions on the unlabeled data set first. Personnel give priority to labeling samples that the model predicts are uncertain, gradually guiding the model to learn, and reducing invalid labeling. With the accumulation of labeled data, the model is continuously trained and updated. After the model performance is improved, the data set is predicted again to identify new samples to be labeled, forming an iterative labeling process; according to the model obtained by the flame detection model training, the presence of flames is detected, and multiple consecutive frames of fire trajectory images are numbered in sequence and put into the folder '1'. Multiple consecutive frames of flame-free images and interference images such as light source images, tungsten filament lamp images, and reflective clothing images are numbered in sequence and put into the folder '0'. During training, 30 consecutive images are randomly selected, and the channels are 3 channels of RGB. The images are scaled to a uniform size of 128*128 to form a 30*128*128*3 matrix. These data are used as input data, and the output is 2 classifications.
[0112] Through training and learning, LSTM learned to extract flame features of continuous pictures from continuous images, and its stability and robustness were significantly improved.
[0113] The flame detection model is deployed to perform multi-frame sequence image LSTM model classification: the model is deployed to the fire identification system, the images collected by the surveillance camera are used to detect suspected flame areas, and multiple consecutive frame samples of the suspected flame area are put into the multi-frame sequence LSTM classification network to determine whether flames occur in the area.
[0114] In one implementation, after obtaining the flame detection result of whether the flame trajectory is a real flame phenomenon, the method further includes:
[0115] When the flame trajectory is a real flame phenomenon, obtaining the regional coordinate information of the image data information corresponding to the flame trajectory, and judging the fire level according to the flame trajectory;
[0116] If the fire level is a minor fire, the local fire extinguishing device is automatically triggered;
[0117] If the fire level is medium, an alarm message will be automatically sent to the backend;
[0118] If the fire level is a serious fire, an alarm will be automatically sounded.
[0119] In this implementation, once smoking is detected in the current area, the system will immediately sound an alarm, which is generally a sound alarm. It plays a pre-recorded alarm voice and pushes relevant information to the current area manager. The information push system transmits data (image information and specific location captured by the monitoring) to the rear system through the http network, making it convenient for the area manager to intervene quickly.
[0120] Aiming at the problems that the existing flame detection methods have slow response, are easily affected by the environment, and have low detection accuracy due to misjudgment, this paper proposes a flame detection method based on LSTM multi-frame sequence. The overall framework diagram is shown in Figure 3 As shown, the method comprises the following parts: data collection, flame trajectory generation, image multi-frame sequence LSTM classification model training, model deployment, multi-frame sequence image LSTM model classification, alarm triggering and information push, by first performing flame potential area detection on the image in the monitoring video, and then using tracking algorithms such as ByteTrack to obtain a number of flame area trajectories, and then inputting the image of each flame trajectory into the flame detection model constructed based on the LSTM multi-frame sequence neural network to confirm whether it is a flame, and can detect the real flame dynamics from a series of continuous frames, even in a complex background, it can reduce false alarms, which is conducive to improving the accuracy of flame detection. The method of the present invention can quickly and accurately detect flames in a complex environment.
[0121] Embodiment 2:
[0122] The present invention provides a flame detection system based on LSTM multi-frame sequence, the structural composition diagram is as follows Figure 4 As shown, including:
[0123] Data acquisition module: used to obtain image data information of the area to be identified;
[0124] Trajectory generation module: used for generating flame trajectory by using ByteTrack algorithm based on the image data information corresponding to the potential flame area when there is a potential flame area in the image data information;
[0125] Flame detection module: used to obtain a flame detection result of whether the flame trajectory is a real flame phenomenon based on the flame trajectory using a pre-built flame detection model;
[0126] The flame detection model in the flame detection module is obtained by training the LSTM multi-frame sequence neural network using historical flame trajectories as input of training data and labeled data corresponding to the historical flame trajectories as output of training data.
[0127] The potential flame area in the above trajectory generation module includes the following judgment process:
[0128] According to a preset flame image database, using a structural similarity index method, calculating the similarity between the data in the flame image database and the image data information;
[0129] When the similarity is greater than a preset similarity threshold, it is determined that there is a potential flame area in the graphic data information.
[0130] The above-mentioned trajectory generation module generates flame trajectories based on the image data information corresponding to the potential flame area using the ByteTrack algorithm, including:
[0131] Perform image preprocessing according to the image data information corresponding to the potential flame area to obtain processed image data;
[0132] Perform feature extraction based on the processed image data to obtain corresponding image feature data;
[0133] Automatically annotating each flame object in the image feature data to obtain a bounding box and a confidence score of each flame object;
[0134] Generate a flame track using a ByteTrack algorithm based on the bounding box of each flame object whose confidence score is greater than a set confidence threshold;
[0135] Wherein, the image preprocessing includes: data denoising, data enhancement and color space conversion;
[0136] The image feature data includes: color feature, texture feature, shape feature and temperature feature.
[0137] In the above trajectory generation module, based on the bounding box of each flame object whose confidence score is greater than the set confidence threshold, the flame trajectory is generated using the ByteTrack algorithm, including:
[0138] According to the bounding boxes of each flame object whose confidence score is greater than a set confidence threshold, the overlap between the bounding boxes is calculated using the ByteTrack algorithm to perform bounding box matching;
[0139] According to the coordinate information of the successfully matched bounding box in each frame, the center points or key points of the successfully matched bounding box are connected to form a continuous trajectory line;
[0140] The continuous trajectory lines are superimposed and displayed to generate a flame trajectory.
[0141] The flame detection model in the above-mentioned flame detection module includes the following construction process:
[0142] Acquire a continuous historical video sequence, which includes actual flame trajectories and false flame trajectories of the flame;
[0143] Extract frames from the historical video sequence, and for each frame or selected key frame, accurately mark the flame area using automated tools, including but not limited to bounding box positioning, pixel-level segmentation or key point marking, to generate corresponding annotation data;
[0144] The actual flame trajectories and false flame trajectories corresponding to the labeled data of adjacent frames are organized into multi-frame sequences in chronological order, and the flame position information of multiple consecutive frames in each sequence is used as the input of training data;
[0145] Outputting the labeled data corresponding to the flame position information as training data;
[0146] Based on the input and output of the training data, the LSTM multi-frame sequence neural network is trained, and a cross entropy loss function is used to optimize the network weights of the LSTM multi-frame sequence neural network to obtain a flame detection model.
[0147] After the flame detection module obtains the flame detection result of whether the flame trajectory is a real flame phenomenon, it also includes:
[0148] When the flame trajectory is a real flame phenomenon, obtaining the regional coordinate information of the image data information corresponding to the flame trajectory, and judging the fire level according to the flame trajectory;
[0149] If the fire level is a minor fire, the local fire extinguishing device is automatically triggered;
[0150] If the fire level is medium, an alarm message will be automatically sent to the backend;
[0151] If the fire level is a serious fire, an alarm will be automatically sounded.
[0152] Embodiment 3:
[0153] Based on the same inventive concept, the present invention also provides a computer device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function, so as to implement the steps of a flame detection method based on LSTM multi-frame sequence in the above embodiment.
[0154] Embodiment 4:
[0155] Based on the same inventive concept, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the computer device and the extended storage medium supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the steps of a flame detection method based on LSTM multi-frame sequence in the above embodiment.
[0156] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.
[0157] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0158] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0160] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A flame detection method based on LSTM multi-frame sequence, characterized in that: include: Obtaining image data information of the area to be identified; When there is a potential flame area in the image data information, generating a flame track using a ByteTrack algorithm based on the image data information corresponding to the potential flame area; Based on the flame trajectory, using a pre-built flame detection model, a flame detection result is obtained to determine whether the flame trajectory is a real flame phenomenon; The flame detection model is obtained by training the LSTM multi-frame sequence neural network using historical flame trajectories as input of training data and labeled data corresponding to the historical flame trajectories as output of training data.
2. The method according to claim 1, characterized in that The potential flame area includes the following judgment process: According to a preset flame image database, using a structural similarity index method, calculating the similarity between the data in the flame image database and the image data information; When the similarity is greater than a preset similarity threshold, it is determined that there is a potential flame area in the graphic data information.
3. The method according to claim 1, characterized in that The generating of the flame track by using the ByteTrack algorithm based on the image data information corresponding to the potential flame area includes: Perform image preprocessing according to the image data information corresponding to the potential flame area to obtain processed image data; Perform feature extraction based on the processed image data to obtain corresponding image feature data; Automatically annotating each flame object in the image feature data to obtain a bounding box and a confidence score of each flame object; Generate a flame track using a ByteTrack algorithm based on the bounding box of each flame object whose confidence score is greater than a set confidence threshold; Wherein, the image preprocessing includes: data denoising, data enhancement and color space conversion; The image feature data includes: color feature, texture feature, shape feature and temperature feature.
4. The method according to claim 3, characterized in that The flame track is generated based on the bounding box of each flame object whose confidence score is greater than the set confidence threshold, using the ByteTrack algorithm, including: According to the bounding boxes of each flame object whose confidence score is greater than a set confidence threshold, the overlap between the bounding boxes is calculated using the ByteTrack algorithm to perform bounding box matching; According to the coordinate information of the successfully matched bounding box in each frame, the center points or key points of the successfully matched bounding box are connected to form a continuous trajectory line; The continuous trajectory lines are superimposed and displayed to generate a flame trajectory.
5. The method according to claim 1, characterized in that The flame detection model includes the following construction process: Acquire a continuous historical video sequence, which includes actual flame trajectories and false flame trajectories of the flame; Extract frames from the historical video sequence, and for each frame or selected key frame, accurately mark the flame area using automated tools, including but not limited to bounding box positioning, pixel-level segmentation or key point marking, to generate corresponding annotation data; The actual flame trajectories and false flame trajectories corresponding to the labeled data of adjacent frames are organized into multi-frame sequences in chronological order, and the flame position information of multiple consecutive frames in each sequence is used as the input of training data; Outputting the labeled data corresponding to the flame position information as training data; Based on the input and output of the training data, the LSTM multi-frame sequence neural network is trained, and a cross entropy loss function is used to optimize the network weights of the LSTM multi-frame sequence neural network to obtain a flame detection model.
6. The method according to claim 1, characterized in that After obtaining the flame detection result of whether the flame trajectory is a real flame phenomenon, the method further includes: When the flame trajectory is a real flame phenomenon, obtaining the regional coordinate information of the image data information corresponding to the flame trajectory, and judging the fire level according to the flame trajectory; If the fire level is a minor fire, the local fire extinguishing device is automatically triggered; If the fire level is medium, an alarm message will be automatically sent to the backend; If the fire level is a serious fire, an alarm will be automatically sounded.
7. A flame detection system based on LSTM multi-frame sequence, characterized in that: include: Data acquisition module: used to obtain image data information of the area to be identified; Trajectory generation module: used for generating flame trajectory by using ByteTrack algorithm based on the image data information corresponding to the potential flame area when there is a potential flame area in the image data information; Flame detection module: used to obtain a flame detection result of whether the flame trajectory is a real flame phenomenon based on the flame trajectory using a pre-built flame detection model; The flame detection model in the flame detection module is obtained by training the LSTM multi-frame sequence neural network using historical flame trajectories as input of training data and labeled data corresponding to the historical flame trajectories as output of training data.
8. The system according to claim 7, characterized in that The trajectory generation module generates a flame trajectory based on the image data information corresponding to the potential flame area using the ByteTrack algorithm, including: Perform image preprocessing according to the image data information corresponding to the potential flame area to obtain processed image data; Perform feature extraction based on the processed image data to obtain corresponding image feature data; Automatically annotating each flame object in the image feature data to obtain a bounding box and a confidence score of each flame object; Generate a flame track using a ByteTrack algorithm based on the bounding box of each flame object whose confidence score is greater than a set confidence threshold; Wherein, the image preprocessing includes: data denoising, data enhancement and color space conversion; The image feature data includes: color feature, texture feature, shape feature and temperature feature.
9. A computer device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, a flame detection based on LSTM multi-frame sequence as described in any one of claims 1 to 6 is implemented.
10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed, a flame detection based on LSTM multi-frame sequence as described in any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Security scene flame detection method based on deep learning
CN111814638A
Time sequence fire identification method and system based on single-step detector
CN113052226A
Cited By
Continuous flame smoke recognition method, electronic equipment and storage medium
CN120823555A
Liquid rocket engine combustion tail flame detection method
CN122530770A
