Video stream fire and smoke detection method, device and medium for fire and smoke protection

Abstract

The present application discloses a method, device and medium for video stream fire detection for fire protection, the method comprising: based on a preset front-end monitoring device, obtaining video stream data corresponding to a to-be-detected area, and decoding the video stream data into an RGB image; inputting the RGB image into a trained fire detection model for primary detection to identify at least one suspicious area corresponding to the to-be-detected area, and outputting corresponding suspicious area information; issuing a detection command to a front-end monitoring device corresponding to the suspicious area according to coordinate information in the suspicious area information, so as to perform secondary detection on the suspicious area, and determine a target area with fire risk; screening the location information of the target area, and performing secondary screening of feature information on the screened static pseudo-target to be determined, so as to upload the alarm information corresponding to the screened and confirmed fire target.

Description

Technical Field

[0001] The present application relates to the technical field of smoke and fire detection, and in particular to a method, device and medium for detecting smoke and fire in a video stream for smoke and fire protection. Background Art

[0002] In today's rapidly developing urbanization process, urban security and forest protection, as core components of the modern urban security management system, are becoming increasingly important. With the advancement of science and technology and the application of intelligent technology, these two fields are undergoing unprecedented changes, jointly building an efficient and intelligent security network. As a densely populated area with frequent economic activities, the safety of cities is not only related to the safety of citizens' lives and property, but also the foundation of social stability and harmony. As an important part of the earth's ecosystem, forest protection plays an irreplaceable role in maintaining ecological balance and preventing natural disasters.

[0003] Traditionally, urban security and forest protection mainly rely on physical means such as manual patrols and temperature / smoke sensors for monitoring. However, these traditional methods have many limitations. Although temperature sensors and smoke sensors can achieve fire warnings to a certain extent, their sensor performance is easily affected by external factors such as ambient temperature and humidity, resulting in high false alarm and missed alarm rates, affecting the accuracy and reliability of warnings. In addition, these traditional sensors mostly use single-chip microcomputers and simple circuit components, and their stability and durability are difficult to guarantee under long-term use, and the maintenance cost is high.

[0004] With the rapid development of computer vision and artificial intelligence technology, especially the widespread application of deep learning technology, fire and smoke detection technology has ushered in new breakthroughs. The target detection algorithm based on deep learning can accurately identify the smoke and fire characteristics at the beginning of a fire by analyzing the dynamic changes in video images, and achieve fast and accurate early warning. This technology not only overcomes the shortcomings of traditional sensors that are easily affected by environmental interference, but also greatly improves the accuracy and stability of detection.

[0005] However, despite the remarkable progress made in deep learning-based fire and smoke detection technology, there are still some challenges in practical applications. In particular, in complex and changing natural environments, such as cloud occlusion, dense fog, mottled tree shade, pool reflections, and interference from static pseudo-targets such as white landmarks, the fire and smoke detection algorithm of a single-frame image stream is prone to generate a large number of false alarms, which brings a huge burden to subsequent processing, not only wasting precious human and material resources, but also affecting the overall efficiency and credibility of the system. Summary of the invention

[0006] The embodiments of the present application provide a video stream smoke and fire detection method, device and medium for smoke and fire protection, which are used to solve the above-mentioned technical problems.

[0007] On the one hand, an embodiment of the present application provides a video stream smoke and fire detection method for smoke and fire protection, comprising:

[0008] Based on the preset front-end monitoring device, the video stream data corresponding to the area to be detected is obtained, and the video stream data is decoded into an RGB image;

[0009] Inputting the RGB image into a trained fireworks detection model for initial detection to identify at least one suspicious area corresponding to the area to be detected, and outputting corresponding suspicious area information;

[0010] According to the coordinate information in the suspicious area information, a detection command is issued to the front-end monitoring device corresponding to the suspicious area to perform a secondary detection on the suspicious area and determine the target area with fireworks risk; the detection command includes: a movement command and a zoom command;

[0011] The position information of the target area is screened, and the feature information of the screened pending static pseudo targets is screened again, so as to upload the alarm information corresponding to the fireworks targets that have been screened and confirmed.

[0012] In one implementation of the present application, the location information of the target area is screened, specifically including:

[0013] Determine the two adjacent frame detection frames within the specified time period, and construct a corresponding IoU cost matrix for the IoU of the previous frame detection frame and the next frame detection frame in the two adjacent frame detection frames;

[0014] By using the Hungarian algorithm, the detection frame of the previous frame is compared with the detection frame of the next frame to determine whether the detection frame of the previous frame matches the detection frame of the next frame;

[0015] If so, it is determined that the target area is a pending static pseudo target, and if not, it is determined that there is a dynamic target in the target area.

[0016] In one implementation of the present application, the specific calculation formula of the IoU cost matrix is:

[0017]

[0018] Among them, A i For the tth 1 The i-th box among the m detection boxes at the moment, B j For the tth 2 The jth box among the n detection boxes at the moment, A i ∩B j Indicates A i With B j The intersection of the areas, A i ∪Bj Indicates A i With B j The union of the areas.

[0019] In one implementation of the present application, the feature information of the selected pending static pseudo targets is secondary screened, specifically including:

[0020] Obtain multiple detection frames within a specified time period, and for each detection frame, perform a cropping operation on the corresponding RGB image according to the position information of the bounding box corresponding to the detection frame;

[0021] The cropped RGB image is input into a pre-trained RID network to extract the feature vector corresponding to the pending static pseudo target, and the target cosine distance between the feature vectors of the two adjacent frames in the multi-frame detection frame is calculated;

[0022] Comparing the target cosine distance with a preset threshold value to perform a secondary determination on the pending static pseudo target;

[0023] If the target cosine distance is less than the preset threshold, the pending static pseudo target is determined to be a static pseudo target; if the target cosine distance is greater than the preset threshold, the pending static pseudo target is determined to be a real fireworks target.

[0024] In one implementation of the present application, calculating the target cosine distance between feature vectors of two adjacent frame detection frames in the multi-frame detection frame specifically includes:

[0025] Calculating the cosine similarity between the feature vectors of two adjacent frame detection frames in the multi-frame detection frames, so as to determine the corresponding target cosine distance according to the cosine similarity;

[0026] The specific calculation formula of the cosine similarity is:

[0027]

[0028] Among them, A i and B i The feature vector extracted from the detection boxes at two different moments.

[0029] In one implementation of the present application, according to the coordinate information in the suspicious area information, a detection command is issued to the front-end monitoring device corresponding to the suspicious area to perform a secondary detection on the suspicious area and determine the target area with fireworks risk, specifically including:

[0030] According to the coordinate information of the designated position in the suspicious area information, a movement command is sent to the pan / tilt camera of the area to be detected to control the pan / tilt camera to turn to the suspicious area;

[0031] Sending a zoom command to the pan / tilt camera to control the pan / tilt camera to zoom in on the suspicious area so that the center position of the pan / tilt camera is consistent with the center position of the suspicious area;

[0032] The suspicious area is continuously detected within a preset time period, and when the number of detected targets within the preset time period is greater than a preset target appearance threshold, a target area with a fire and smoke risk in the area to be detected is determined.

[0033] In one implementation of the present application, based on a preset front-end monitoring device, video stream data corresponding to the area to be detected is obtained, and the video stream data is decoded into an RGB image, specifically including:

[0034] Deploy the PTZ camera and edge computing module to a preset position, and configure corresponding inspection points for the PTZ camera according to actual detection requirements; the preset position is used to represent the high point position in the city or forest;

[0035] Determine the RTSP address of the PTZ camera and the information receiving address corresponding to the alarm information, and configure the RTSP address and the information receiving address in the edge computing module;

[0036] Based on the pan-tilt camera, the video stream data of the area to be detected is collected, and the video encoding format corresponding to the video stream data is determined;

[0037] Pull the video stream data according to the RTSP address and through the edge computing module;

[0038] According to the video encoding format, a decoder corresponding to the video stream data is determined, and the video stream data is decoded by the decoder to obtain a corresponding RGB image.

[0039] In one implementation of the present application, the RGB image is input into a trained fire and smoke detection model for initial detection to identify at least one suspicious area corresponding to the area to be detected, and the corresponding suspicious area information is output, specifically including:

[0040] Constructing a smoke and fire detection model, and setting a corresponding detection threshold for the smoke and fire detection model;

[0041] Determine a sample image with labeled suspicious area information, and input the sample image into the fireworks detection model for training until the target suspicious area information output by the fireworks detection model matches the labeled suspicious area information, thereby obtaining a trained fireworks detection model;

[0042] Inputting the RGB image into the trained fireworks detection model to perform an initial detection on the RGB image and determine at least one suspicious area;

[0043] The suspicious area information corresponding to each suspicious area is outputted through the fireworks detection model; the suspicious area information includes the length, width, detection category, detection category probability and coordinate information of the designated position corresponding to the suspicious area.

[0044] On the other hand, an embodiment of the present application further provides a video stream smoke and fire detection device for smoke and fire protection, the device comprising:

[0045] at least one processor;

[0046] and, a memory communicatively coupled to the at least one processor;

[0047] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned video stream smoke and fire detection method for smoke and fire protection.

[0048] On the other hand, an embodiment of the present application further provides a non-volatile computer storage medium storing computer executable instructions, which, when executed, implements the above-mentioned video stream smoke and fire detection method for smoke and fire protection.

[0049] The present application provides a video stream smoke and fire detection method, device and medium for smoke and fire protection, which at least have the following beneficial effects:

[0050] The video stream data of the area to be detected is obtained in real time through the front-end monitoring equipment and decoded into RGB images, providing a high-quality data foundation for subsequent fireworks detection. The trained fireworks detection model is used for initial detection, which can quickly identify suspicious areas and greatly improve the detection efficiency. The secondary detection command (move and zoom) is used to observe the suspicious area in more detail, which can flexibly respond to the fireworks detection needs in different locations and situations, further improving the accuracy of fireworks detection. The location information of the target area is screened, and by comparing the IoU of the detection frame of the previous and next frames, static pseudo-targets caused by camera jitter, light changes, etc. are effectively excluded, reducing the false alarm rate. The feature information of the selected static pseudo-targets is secondary screened, and the cosine distance of the feature vector is used to determine whether it is a real fireworks target, further reducing the possibility of false alarms. Through real-time monitoring and alarm, fireworks risks can be discovered and handled in a timely manner, effectively preventing the spread and expansion of the fire. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0052] Figure 1 A flowchart of a method for detecting fireworks in a video stream for fireworks protection provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of the process of fire and smoke detection provided in an embodiment of the present application;

[0054] Figure 3 A flowchart of a smoke and fire detection algorithm provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of the internal structure of a video stream smoke and fire detection device for smoke and fire protection provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0057] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0058] Figure 1 A flowchart of a method for detecting fireworks in a video stream for fireworks protection provided in an embodiment of the present application.

[0059] The analysis method involved in the embodiments of the present application can be implemented by a terminal device or a server, and the present application does not impose any special restrictions on this. For the convenience of understanding and description, the following embodiments are described in detail by taking a server as an example.

[0060] It should be noted that the server may be a single device or a system consisting of multiple devices, that is, a distributed server, and this application does not make any specific limitation on this.

[0061] like Figure 1 As shown, a video stream smoke and fire detection method for smoke and fire protection provided in an embodiment of the present application includes:

[0062] 101. Based on a preset front-end monitoring device, obtain video stream data corresponding to the area to be detected, and decode the video stream data into an RGB image.

[0063] Specifically, in one embodiment of the present application, based on a preset front-end monitoring device, obtaining video stream data corresponding to the area to be detected, and decoding the video stream data into an RGB image specifically includes:

[0064] Deploy the PTZ camera and edge computing module to the preset position, and configure the corresponding inspection points for the PTZ camera according to the actual detection needs; the preset position is used to indicate the high point position in the city or forest;

[0065] Determine the RTSP address of the PTZ camera and the information receiving address corresponding to the alarm information, and configure the RTSP address and information receiving address in the edge computing module;

[0066] Collect video stream data of the area to be detected based on the pan-tilt camera, and determine the video encoding format corresponding to the video stream data;

[0067] Pull video stream data based on the RTSP address and through the edge computing module;

[0068] According to the video encoding format, the decoder corresponding to the video stream data is determined, and the video stream data is decoded by the decoder to obtain the corresponding RGB image.

[0069] In one embodiment, in order to build an efficient urban or forest fire monitoring system, a technical solution combining a PTZ camera and an edge computing module is adopted. First, a PTZ camera and an edge computing module are deployed on the top of a high-rise building in the city or at a high point in the forest. These locations can provide a wide field of view and facilitate monitoring of a large area. According to actual detection needs, a series of inspection points are configured for the PTZ camera. These points cover key areas in the city or forest, such as flammable material storage points, crowded areas, etc. The PTZ camera will automatically rotate to these points for shooting according to the preset inspection plan.

[0070] Determine the RTSP (Real Time Streaming Protocol) address of the PTZ camera, which is the network protocol address used to pull video stream data. At the same time, determine the information receiving address corresponding to the alarm information so that the alarm information can be sent in time when fireworks are detected. Configure the above two addresses in the edge computing module to ensure that the module can correctly pull the video stream data from the PTZ camera and send alarm information when necessary.

[0071] The PTZ camera starts to collect video stream data of the area to be detected, which is compressed and transmitted in a specific video encoding format. The edge computing module or related tools determine the specific encoding format of the video stream data, such as H.264, H.265, etc. This is key information for the subsequent decoding process. The edge computing module actively initiates a request to the PTZ camera based on the configured RTSP address to pull the video stream data. These data are transmitted to the edge computing module through the network for processing.

[0072] According to the previously determined video encoding format, the edge computing module selects the corresponding decoder. The decoder is a software or hardware module specifically used to decode the compressed video stream data into raw image data. The decoder decodes the pulled video stream data to obtain the RGB image of each frame. These images are the basis for subsequent smoke and fire detection algorithm processing.

[0073] In one embodiment, the front-end video surveillance equipment is deployed, and the PTZ camera and the edge computing module storing the video stream smoke and fire detection algorithm are deployed to high points in cities and forests. The inspection points of the PTZ camera are configured according to actual needs, and the RTSP address of the PTZ camera and the alarm information receiving address of the video surveillance center are set in the edge computing module.

[0074] The edge computing module pulls the real-time video stream and decodes the video stream. The edge computing module uses the gstreamer framework to obtain the real-time video data of the PTZ camera through the RTSP address, and further decodes it into an RGB image. Among them, gstreamer is a powerful multimedia framework. According to the video encoding format provided by the PTZ camera (such as H.264, H.265, etc.), select the appropriate GStreamer decoder to decode the video stream. For example, for the H.264-encoded video stream, this application uses the omxh264dec (for NVIDIA hardware acceleration) decoder to decode the video stream and finally obtain the corresponding RGB image.

[0075] 102. Input the RGB image into a trained fireworks detection model for initial detection to identify at least one suspicious area corresponding to the area to be detected, and output corresponding suspicious area information.

[0076] Specifically, in one embodiment of the present application, the RGB image is input into a trained fire and smoke detection model for initial detection to identify at least one suspicious area corresponding to the area to be detected, and the corresponding suspicious area information is output, specifically including:

[0077] Build a fireworks detection model and set the corresponding detection threshold for the fireworks detection model;

[0078] Determine a sample image with labeled suspicious area information, and input the sample image into a fireworks detection model for training until the target suspicious area information output by the fireworks detection model matches the labeled suspicious area information, thereby obtaining a trained fireworks detection model;

[0079] Input the RGB image into the trained fireworks detection model to perform an initial detection on the RGB image and determine at least one suspicious area;

[0080] The suspicious area information corresponding to each suspicious area is outputted through the fireworks detection model; the suspicious area information includes the length, width, detection category, detection category probability and coordinate information of the designated position corresponding to the suspicious area.

[0081] In one embodiment, first, a deep learning framework (such as TensorFlow, PyTorch, etc.) is selected to build a fireworks detection model. The model can be based on a convolutional neural network (CNN) architecture, such as YOLO (You Only LookOnce), SSD (Single Shot MultiBox Detector) or Faster R-CNN, which perform well in target detection tasks. According to the needs of the actual application scenario, the model structure is appropriately adjusted, such as increasing or decreasing the number of network layers, adjusting the size of the convolution kernel, etc.

[0082] Next, set an appropriate detection threshold for the fireworks detection model. This threshold is used to determine whether a certain area is considered a "suspicious area" when the model predicts. Usually, this threshold is set based on the performance indicators of the model (such as accuracy, recall rate) and the tolerance for false positives and false negatives in practical applications. For example, the detection threshold can be set to 0.5, which means that only when the model predicts that a certain area is a fireworks category with a probability of more than 50%, the area is considered a suspicious area.

[0083] Collect a large number of RGB images with and without fireworks as training data sets. For each image, use annotation tools (such as LabelImg) to manually annotate all suspicious areas (i.e., where fireworks appear), and record the length, width, category (fireworks), and coordinate information of these areas. This annotation information will be used as a supervisory signal for model training.

[0084] Input the labeled sample images into the fireworks detection model for training. During the training process, the model will learn how to extract features from the image and predict the suspicious areas in the image based on these features. By continuously adjusting the model parameters, the target suspicious area information output by the model is made to match the manually labeled suspicious area information as closely as possible. When the performance of the model on the validation set meets expectations, the model is considered to be trained.

[0085] Input the RGB image to be detected into the trained fireworks detection model. The model will scan the image pixel by pixel or region by region, and use the learned feature extraction and classification capabilities to identify all areas in the image that may contain fireworks and mark them as suspicious areas. For each area identified as suspicious, the fireworks detection model will output a series of detailed information.

[0086] The length is used to indicate the horizontal size of the suspicious area; the width is used to indicate the vertical size of the suspicious area; the detection category is used to indicate "fireworks" in this embodiment; the detection category probability is used to indicate the confidence of the model in predicting that the area is fireworks, usually expressed as a percentage; the coordinate information of the specified position is used to indicate the specific position of the suspicious area in the image, and the specified position in this embodiment refers to the upper left corner of the image.

[0087] In one embodiment, the edge computing module inputs the acquired RGB image into the smoke and fire detection model for initial detection, where the smoke and fire detection model uses the YOLOV8s model as the benchmark model. In order to increase the probability of detecting suspicious areas and reduce the problem of missing suspicious targets due to long-distance inspections by PTZ cameras, a lower detection threshold P is usually set at this stage. 1 .

[0088] In the initial detection stage, an input RGB image may contain multiple suspicious areas at the same time, denoted as R = {r 1 ,r 2 ,...,r n}, where r i represents the i-th detected suspicious area, and any suspicious area contains information r i =[x i ,y i ,w i ,h i ,c i ,p i ], where x i ,y i represents the coordinate information of the upper left corner of the i-th detected suspicious area, w i ,h i Indicates the width and height of the detected suspicious area, c i represents the detected category, p i Represents the probability of detected category.

[0089] 103. According to the coordinate information in the suspicious area information, a detection command is issued to the front-end monitoring device corresponding to the suspicious area to perform a secondary detection on the suspicious area and determine the target area with fireworks risk.

[0090] It should be noted that the detection commands in the embodiment of the present application include: movement commands and zoom commands.

[0091] Specifically, in one embodiment of the present application, according to the coordinate information in the suspicious area information, a detection command is issued to the front-end monitoring device corresponding to the suspicious area to perform a secondary detection on the suspicious area and determine the target area with fireworks risk, specifically including:

[0092] According to the coordinate information of the specified position in the suspicious area information, a movement command is sent to the PTZ camera in the area to be detected to control the PTZ camera to turn to the suspicious area;

[0093] Sending a zoom command to the PTZ camera to control the PTZ camera to zoom in on the suspicious area so that the center position of the PTZ camera is consistent with the center position of the suspicious area;

[0094] The suspicious area is continuously detected within a preset time period, and when the number of detected targets within the preset time period is greater than a preset target appearance threshold, a target area with a fire and smoke risk in the area to be detected is determined.

[0095] In one embodiment, the edge computing module performs a second continuous detection on the amplified suspicious area. The second continuous detection goal is to determine whether there are still multiple target detections within the specified time T. When the number of detected targets is greater than the threshold N, it is determined that there is a detection target in the suspicious area and enters the third target confirmation stage. Otherwise, jump out of this suspicious area and perform a secondary inspection of the next suspicious area.

[0096] In one embodiment, an intelligent video surveillance system is integrated with advanced fire and smoke detection technology to monitor fire and smoke risks in forests, factories, residential areas and other places in real time. The system captures live video through a pan-tilt camera and analyzes the video content in combination with a fire and smoke detection algorithm. First, through other monitoring means (such as thermal imaging, smoke detectors, etc.) or manual reports, information about suspicious areas that may have fire and smoke risks in a certain area to be detected (such as a certain area in the forest) is obtained. The information contains the precise coordinates of the suspicious area (such as longitude and latitude or relative coordinates relative to a reference point).

[0097] According to the coordinates of the received suspicious area information, the system sends a movement command to the PTZ camera in the area to be detected. The PTZ camera adjusts its horizontal and vertical angles according to the command to ensure that the camera's field of view can cover the suspicious area. In order to observe the suspicious area more clearly, the system then sends a zoom command to the PTZ camera. The camera adjusts its lens focal length according to the command so that the suspicious area is magnified in the image, ensuring that the center position of the PTZ camera is consistent with the center position of the suspicious area, thereby obtaining a high-definition image of the suspicious area.

[0098] In the next preset time period (such as 5 consecutive minutes), the system continuously captures video frames and detects fireworks in the suspicious area. Each frame of the image is processed by the fireworks detection algorithm to identify whether there are fireworks features. The number of times the target (i.e., fireworks features) is detected in the preset time period is counted. If the number of detected targets is greater than the preset target occurrence threshold (such as 3 times), the system determines that there is a target area with fireworks risk in the area to be detected. This means that there is a high probability that fireworks do exist in the suspicious area, and further measures need to be taken immediately.

[0099] 104. Screen the location information of the target area, and perform secondary screening on the feature information of the screened pending static pseudo targets, so as to upload the alarm information corresponding to the screened confirmed fireworks targets.

[0100] Specifically, in one embodiment of the present application, the location information of the target area is screened, including:

[0101] Determine the two adjacent frame detection frames within the specified time period, and construct the corresponding IoU cost matrix for the IoU of the previous frame detection frame and the next frame detection frame in the two adjacent frame detection frames;

[0102] The detection frame of the previous frame is compared with the detection frame of the next frame by using the Hungarian algorithm to determine whether the detection frame of the previous frame matches the detection frame of the next frame.

[0103] If so, it is determined that the target area is a pending static pseudo target, and if not, it is determined that there is a dynamic target in the target area.

[0104] In one embodiment, when the edge computing module determines the suspicious area r i After the target is found, the third target confirmation is required. The main purpose of this stage is to check for false fireworks alarms caused by various static false target misleading models. The third confirmation stage mainly relies on two aspects of evaluation criteria, namely location information and feature information screening.

[0105] Position information screening, obtain the multi-frame data detected in the sixth stage within the time T and perform the following processing. The present invention constructs a cost matrix for the IoU of each two frames of detection frames within the time T, and uses the cost matrix to determine whether the previous and next detection frames exist continuously. The specific implementation method is as follows. The present invention constructs an IoU matrix for the detection frame of each frame and the detection frame of the next frame. Each element in the matrix represents the IoU value between the detection frames of the previous and next frames. For example, if t i There are m detection boxes and t at the moment i+1 If there are n detection frames at a time, an mxn IoU matrix will be formed. Next, the Hungarian algorithm is used to match the detection frames of the previous and next frames. i+1 All check boxes can be compared with ti If the detection frame at the moment matches, the target in the suspicious area is temporarily determined to be a static pseudo target, otherwise it is determined to be a dynamic target. Since static pseudo targets do not change in form like smoke and flames during continuous detection, it can be assumed that the constantly changing dynamic targets are real fireworks targets, and the alarm information is directly transmitted back. For pseudo targets temporarily determined to be static, they need to be input into the feature information screening module for secondary judgment.

[0106] In one embodiment of the present application, the specific calculation formula of the IoU cost matrix is:

[0107]

[0108] Among them, A i For the tth 1 The i-th box among the m detection boxes at the moment, B j For the tth 2 The jth box among the n detection boxes at the moment, A i ∩B j Indicates A i With B j The intersection of the areas, A i ∪B j Indicates A i With B j The union of the areas.

[0109] In one embodiment of the present application, the secondary screening of feature information of the screened pending static pseudo targets specifically includes:

[0110] Get multiple frames of detection frames within a specified time period, and for each frame of detection frame, perform cropping operations on the corresponding RGB image according to the position information of the bounding box corresponding to the detection frame;

[0111] The cropped RGB image is input into the pre-trained RID network to extract the feature vector corresponding to the pending static pseudo target and calculate the target cosine distance between the feature vectors of the two adjacent frames in the multi-frame detection frame.

[0112] Compare the target cosine distance with the preset threshold to make a secondary judgment on the pending static pseudo target;

[0113] If the target cosine distance is less than the preset threshold, the pending static pseudo target is determined to be a static pseudo target; if the target cosine distance is greater than the preset threshold, the pending static pseudo target is determined to be a real fireworks target.

[0114] In one embodiment, feature information screening, in order to further identify whether the target assumed in process 8 is a static pseudo target, the present invention performs the following operations on the multi-frame detection area, obtains the multi-frame data detected within time T, and cuts the original image according to the detected bounding box position information, and sends it to the pre-trained RID network, and further extracts the feature vector corresponding to the target, which is recorded as: F = {f 1 ,f 2 ,...,f n}, f i ∈R 128 , where f i It is the 128-dimensional feature vector corresponding to the detection area at the i-th moment.

[0115] Since smoke and flame targets are in motion, their detected edge contours should be different, whereas non-smoke and flame data are the same. Therefore, for this target, the cosine distance between the two frames before and after the target within T time is calculated. If the cosine distance is always lower than the set threshold, the target is judged to be a static pseudo target, otherwise it is judged to be a real smoke and flame target.

[0116] In one embodiment of the present application, calculating the target cosine distance between feature vectors of two adjacent frame detection frames in a multi-frame detection frame specifically includes:

[0117] Calculate the cosine similarity between the feature vectors of the two adjacent frame detection frames in the multi-frame detection frame, and determine the corresponding target cosine distance according to the cosine similarity;

[0118] The specific calculation formula of cosine similarity is:

[0119]

[0120] It should be noted that A in the embodiments of the present application i and B i It is the feature vector extracted from two adjacent detection frames at different moments, and the cosine distance = 1-cosine similarity.

[0121] Alarm information is sent back for real fireworks targets, and combined with the above information, it is pushed to patrol personnel in real time, so that emergency rescue can be quickly achieved in the early stage of the fire, thus realizing a closed business loop.

[0122] The above is an embodiment of the method proposed in this application. Based on the same inventive concept, this application embodiment also provides a video stream smoke and fire detection device for smoke and fire protection, and its structure is as follows: Figure 2 shown.

[0123] Figure 2 The flowchart of the smoke and fire detection provided in the embodiment of the present application is shown in FIG. Figure 2As shown, the present application first obtains the video stream data of the pan-tilt camera, and decodes the video stream data into the corresponding RGB image, and then inputs the RGB image into the fireworks detection model for detection, identifies the suspicious area, and then determines the corresponding suspicious area information, so as to upload the alarm information corresponding to the suspicious area and push it to the corresponding inspection personnel, so as to timely discover the fireworks risk and take corresponding measures.

[0124] Figure 3 This is a flow chart of the smoke and fire detection model provided in the embodiment of the present application. Figure 3 As shown, the RGB image is input into the fireworks detection model for initial detection to determine whether there is a suspicious area in the RGB image. If not, other RGB images are input for initial detection. If so, at least one of the identified suspicious areas is checked. Specifically, a movement command and a zoom command are sent to the pan-tilt camera to control the pan-tilt camera to turn to the suspicious area to be checked, and the suspicious area to be checked is zoomed in and focused, so as to perform a second continuous detection to determine whether the suspicious area to be checked can continuously detect fireworks targets. If not, return to the suspicious area screening stage and check other suspicious areas in turn. In the case of continuous detection of fireworks targets, the third target determination is performed on the suspicious area where the fireworks targets are currently detected continuously. First, the location information of the current suspicious area is screened. When t i+1 All check boxes cannot be compared with t i When the detection frame at the time matches, it is determined that there is a dynamic target in the current suspicious area, and the corresponding alarm information is uploaded. i+1 All check boxes can be compared with t i When the detection frame at the moment matches, it is determined to be a static pseudo target, and the feature information of the static pseudo target is further screened, and the cosine distance of the two frames before and after the target in the specified time period is calculated. If the cosine distance is always lower than the preset threshold, the target is judged to be a static pseudo target, otherwise it is judged to be a real fireworks target, and the corresponding alarm information is uploaded.

[0125] Figure 4 The internal structure diagram of a video stream fire detection device for fire protection provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the device includes:

[0126] at least one processor;

[0127] and, a memory communicatively coupled to the at least one processor;

[0128] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to:

[0129] Based on the preset front-end monitoring device, the video stream data corresponding to the area to be detected is obtained, and the video stream data is decoded into an RGB image;

[0130] Input the RGB image into the trained fireworks detection model for initial detection to identify at least one suspicious area corresponding to the area to be detected, and output the corresponding suspicious area information;

[0131] According to the coordinate information in the suspicious area information, a detection command is issued to the front-end monitoring device corresponding to the suspicious area to conduct a secondary detection of the suspicious area and determine the target area with fireworks risk; the detection command includes: a movement command and a zoom command;

[0132] The location information of the target area is screened, and the feature information of the screened pending static pseudo targets is screened again, so that the alarm information corresponding to the fireworks targets that have been screened and confirmed is uploaded.

[0133] The present application also provides a non-volatile computer storage medium storing computer executable instructions. When the computer executable instructions are executed, they can:

[0134] Based on the preset front-end monitoring device, the video stream data corresponding to the area to be detected is obtained, and the video stream data is decoded into an RGB image;

[0135] Input the RGB image into the trained fireworks detection model for initial detection to identify at least one suspicious area corresponding to the area to be detected, and output the corresponding suspicious area information;

[0136] According to the coordinate information in the suspicious area information, a detection command is issued to the front-end monitoring device corresponding to the suspicious area to conduct a secondary detection of the suspicious area and determine the target area with fireworks risk; the detection command includes: a movement command and a zoom command;

[0137] The location information of the target area is screened, and the feature information of the screened pending static pseudo targets is screened again, so that the alarm information corresponding to the fireworks targets that have been screened and confirmed is uploaded.

[0138] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0139] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0140] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0141] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0142] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0145] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0146] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0147] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0148] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0149] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A video stream smoke and fire detection method for smoke and fire protection, characterized in that: The method comprises: Based on the preset front-end monitoring device, the video stream data corresponding to the area to be detected is obtained, and the video stream data is decoded into an RGB image; Inputting the RGB image into a trained fireworks detection model for initial detection to identify at least one suspicious area corresponding to the area to be detected, and outputting corresponding suspicious area information; According to the coordinate information in the suspicious area information, a detection command is issued to the front-end monitoring device corresponding to the suspicious area to perform a secondary detection on the suspicious area and determine the target area with fireworks risk; the detection command includes: a movement command and a zoom command; The position information of the target area is screened, and the feature information of the screened pending static pseudo targets is screened again, so as to upload the alarm information corresponding to the fireworks targets that have been screened and confirmed.

2. The method for video stream smoke and fire detection for smoke and fire protection according to claim 1, characterized in that: Screening the target area for location information includes: Determine the two adjacent frame detection frames within the specified time period, and construct a corresponding IoU cost matrix for the IoU of the previous frame detection frame and the next frame detection frame in the two adjacent frame detection frames; By using the Hungarian algorithm, the detection frame of the previous frame is compared with the detection frame of the next frame to determine whether the detection frame of the previous frame matches the detection frame of the next frame; If so, it is determined that the target area is a pending static pseudo target, and if not, it is determined that there is a dynamic target in the target area.

3. The method for video stream smoke and fire detection for smoke and fire protection according to claim 1, characterized in that: The specific calculation formula of the IoU cost matrix is: Among them, A i is the i-th box among the m detection boxes at time t1, B j is the jth box among the n detection boxes at time t2, A i ∩B j Indicates A i With B j The intersection of the areas, A i ∪B j Indicates A i With B j The union of the areas.

4. The video stream smoke and fire detection method for smoke and fire protection according to claim 1, characterized in that: The feature information of the selected pending static pseudo targets is screened again, including: Obtain multiple detection frames within a specified time period, and for each detection frame, perform a cropping operation on the corresponding RGB image according to the position information of the bounding box corresponding to the detection frame; The cropped RGB image is input into a pre-trained RID network to extract the feature vector corresponding to the pending static pseudo target, and the target cosine distance between the feature vectors of the two adjacent frames in the multi-frame detection frame is calculated; Comparing the target cosine distance with a preset threshold value to perform a secondary determination on the pending static pseudo target; If the target cosine distance is less than the preset threshold, the pending static pseudo target is determined to be a static pseudo target; if the target cosine distance is greater than the preset threshold, the pending static pseudo target is determined to be a real fireworks target.

5. The method for video stream smoke and fire detection for smoke and fire protection according to claim 4, characterized in that: Calculating the target cosine distance between the feature vectors of the two adjacent frame detection frames in the multi-frame detection frame specifically includes: Calculating the cosine similarity between the feature vectors of two adjacent frame detection frames in the multi-frame detection frames, so as to determine the corresponding target cosine distance according to the cosine similarity; The specific calculation formula of the cosine similarity is: Among them, A i and B i The feature vector extracted from the detection boxes at two different moments.

6. The method for video stream smoke and fire detection for smoke and fire protection according to claim 1, characterized in that: According to the coordinate information in the suspicious area information, a detection command is issued to the front-end monitoring device corresponding to the suspicious area to perform a secondary detection on the suspicious area and determine the target area with fireworks risk, specifically including: According to the coordinate information of the designated position in the suspicious area information, a movement command is sent to the pan / tilt camera of the area to be detected to control the pan / tilt camera to turn to the suspicious area; Sending a zoom command to the pan / tilt camera to control the pan / tilt camera to zoom in on the suspicious area so that the center position of the pan / tilt camera is consistent with the center position of the suspicious area; The suspicious area is continuously detected within a preset time period, and when the number of target detections within the preset time period is greater than a preset target appearance threshold, a target area with a fire and smoke risk in the area to be detected is determined.

7. The method for video stream smoke and fire detection for smoke and fire protection according to claim 1, characterized in that: Based on the preset front-end monitoring device, the video stream data corresponding to the area to be detected is obtained, and the video stream data is decoded into an RGB image, specifically including: Deploy the PTZ camera and edge computing module to a preset position, and configure corresponding inspection points for the PTZ camera according to actual detection requirements; the preset position is used to represent the high point position in the city or forest; Determine the RTSP address of the PTZ camera and the information receiving address corresponding to the alarm information, and configure the RTSP address and the information receiving address in the edge computing module; Based on the pan-tilt camera, the video stream data of the area to be detected is collected, and the video encoding format corresponding to the video stream data is determined; Pull the video stream data according to the RTSP address and through the edge computing module; According to the video encoding format, a decoder corresponding to the video stream data is determined, and the video stream data is decoded by the decoder to obtain a corresponding RGB image.

8. The method for video stream smoke and fire detection for smoke and fire protection according to claim 1, characterized in that: The RGB image is input into the trained fire and smoke detection model for initial detection to identify at least one suspicious area corresponding to the area to be detected, and the corresponding suspicious area information is output, which specifically includes: Constructing a smoke and fire detection model, and setting a corresponding detection threshold for the smoke and fire detection model; Determine a sample image with labeled suspicious area information, and input the sample image into the fireworks detection model for training until the target suspicious area information output by the fireworks detection model matches the labeled suspicious area information, thereby obtaining a trained fireworks detection model; Inputting the RGB image into the trained fireworks detection model to perform an initial detection on the RGB image and determine at least one suspicious area; The suspicious area information corresponding to each suspicious area is outputted through the fireworks detection model; the suspicious area information includes the length, width, detection category, detection category probability and coordinate information of the designated position corresponding to the suspicious area.

9. A video stream fire and smoke detection device for fire and smoke protection, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video stream smoke and fire detection method for smoke and fire protection according to any one of claims 1 to 8.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: When the computer executable instructions are executed, the video stream smoke and fire detection method for smoke and fire protection according to any one of claims 1 to 8 is implemented.

Images