Smoke and fire detection method, electronic equipment and storage medium

Abstract

The invention provides a smoke and fire detection method, electronic equipment and a storage medium, and belongs to the field of image processing. The method comprises the steps that a to-be-detected picture is input into a personalized detection model, segmented pictures output by the personalized detection model are obtained, and the personalized detection model is obtained after a pre-trained smoke and fire detection model is trained through user special data; carrying out contour detection and instance separation on the segmentation area of the segmented picture to obtain an effective instance; and under the condition that the effective examples exist, according to the effective examples of which the categories are smoke and / or fire, sending out a smoke and fire alarm. Therefore, the personalized detection model matched with the user special data of the demand scene is customized for the user, so that the personalized detection model represented by the data expected by the user is used for performing smoke and fire identification, and the detection precision and accuracy in different scenes are greatly improved. Meanwhile, smoke and fire detection can be carried out in real time according to pictures shot by the unmanned aerial vehicle, and the response speed is greatly improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a smoke and fire detection method, an electronic device and a storage medium. Background Art

[0002] With the acceleration of urbanization and the continuous improvement of the ecological environment, the risk of fire accidents is also increasing. Therefore, smoke and fire detection for fire prevention and safety management is extremely important in the safety management of modern society.

[0003] Common methods for detecting fireworks include: (1) identifying fireworks by measuring temperature; (2) using drones equipped with sensors to collect images and detect fireworks. Method (1) requires the temperature sensor to be close to the fire area, which affects its response speed to a certain extent and makes it difficult to respond in time. The detection accuracy of method (2) is difficult to meet the needs of different users. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a smoke and fire detection method, an electronic device and a storage medium, which not only improve the response speed of smoke and fire detection, but also improve the detection accuracy of smoke and fire detection to meet the needs of different users.

[0005] In order to achieve the above purpose, the technical solution adopted in this application is as follows:

[0006] In a first aspect, the present application provides a method for detecting smoke and fire, the method comprising:

[0007] Inputting the image to be tested into the personalized detection model to obtain a segmented image output by the personalized detection model; wherein the segmented image includes at least one segmented area and a category of the segmented area, and the personalized detection model is obtained by training a pre-trained fireworks detection model using user-specific data;

[0008] Performing contour detection and instance separation on the segmented areas of the segmented image to obtain valid instances;

[0009] In the case that a valid instance exists, a smoke and fire alarm is issued according to the valid instance.

[0010] In an optional implementation manner, the step of performing contour detection and instance separation on the segmented areas of the segmented image to obtain valid instances includes:

[0011] Smoothing the segmented areas of the category of smoke or fire in the segmented image to obtain a smoothed image; wherein the smoothed image includes at least one smooth connected area; filtering each of the smooth connected areas in the smoothed image to obtain a corrected image; wherein the corrected image includes at least one valid area;

[0012] For each of the valid regions, contour detection and instance separation are performed on the valid region to obtain a valid instance.

[0013] In an optional implementation manner, the step of filtering each of the smooth connected areas in the smooth image to obtain a corrected image includes:

[0014] For each of the smooth connected areas, an area ratio is obtained according to a ratio of an area of ​​the smooth connected area to a total area of ​​the smooth image;

[0015] The smooth connected areas whose area ratio in the smooth image is less than a ratio threshold are filtered out, and the remaining smooth connected areas are used as valid areas to obtain a corrected image.

[0016] In an optional implementation manner, the step of smoothing the segmented area of ​​the category of smoke or fire in the segmented image to obtain a smoothed image includes:

[0017] The segmented areas in the segmented image that belong to the same target category are grouped as the same category; wherein the target category includes smoke and fire;

[0018] An opening operation is performed on each segmented area of ​​the same category group in the segmented image, and then a kernel function of an elliptical kernel is used to process the segmented area to obtain a smooth connected area.

[0019] In an optional implementation, the step of obtaining the personalized detection model includes:

[0020] Use various common samples to iteratively train the initial model to obtain a fireworks detection model;

[0021] Acquire a user-specific data set; wherein the user-specific data includes a plurality of special samples annotated by the user;

[0022] The fireworks detection model is adjusted and trained using the multiple dedicated samples to obtain a personalized detection model.

[0023] In an optional implementation, the step of using the multiple dedicated samples to adjust and train the fireworks detection model to obtain a personalized detection model includes:

[0024] According to the user index, the number of layers of the segmentation head of the fireworks detection model is adjusted to obtain a preliminary adjustment detection model;

[0025] In the case of freezing the feature extraction layer of the preliminary adjustment detection model, using the special sample to iteratively train the preliminary adjustment detection model to obtain an early result model;

[0026] After unfreezing the feature extraction layer of the previous result model and lowering the learning rate, the previous result model is iteratively trained using the dedicated samples to obtain a personalized detection model.

[0027] In an optional implementation, the step of iteratively training the initial model using the general samples to obtain the smoke and fire detection model includes:

[0028] Segment and annotate the fireworks for each general image in the general data set to obtain general samples;

[0029] Randomly cropping and resizing each of the general samples to obtain a general sample set;

[0030] An SPD module and a pkinet module are added to the YOLOv11 neural network to obtain an initial model; wherein the SPD module is used to convert the spatial dimension of the input feature map into a depth dimension, and the pkinet module is used to perform multi-scale convolution and depth-separable convolution on the input feature map;

[0031] The initial model is iteratively trained using the universal sample set to obtain a smoke and fire detection model.

[0032] In an optional implementation manner, the step of issuing a smoke and fire alarm according to the effective example includes:

[0033] Obtaining the location range of the valid instance according to the coverage area corresponding to the image to be tested;

[0034] Based on the position range, an alarm message is issued.

[0035] In a second aspect, the present application provides an electronic device, including a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor can execute the computer program to implement a smoke and fire detection method as described in any one of the aforementioned embodiments.

[0036] In a third aspect, the present application provides a storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the smoke and fire detection method as described in any one of the aforementioned embodiments is implemented.

[0037] In a fourth aspect, the present application provides a smoke and fire detection device, comprising a detection module, a processing module and an alarm module;

[0038] The detection module is used to input the image to be tested into the personalized detection model to obtain a segmented image output by the personalized detection model; wherein the segmented image includes at least one segmented area and a category of the segmented area, and the personalized detection model is obtained by training a pre-trained fireworks detection model using user-specific data;

[0039] The processing module is used to perform contour detection and instance separation on the segmented areas of the segmented image to obtain valid instances;

[0040] The alarm module is used to issue a fire alarm according to a valid instance when there is one.

[0041] The present application provides a method, electronic device and storage medium for detecting fireworks, the method comprising: inputting a picture to be tested into a personalized detection model to obtain a segmented picture output by the personalized detection model, the segmented picture including at least one segmented area and the category of the segmented area, the personalized detection model being obtained by training a pre-trained fireworks detection model using user-specific data; performing contour detection and instance separation on the segmented areas of the segmented picture to obtain valid instances; in the case of the existence of valid instances, issuing a fireworks alarm based on the valid instances classified as smoke and / or fire. In this way, a personalized detection model that matches the user-specific data of the required scenario is customized for the user, so that fireworks recognition is performed using the personalized detection model represented by the user's expected data, which greatly improves the detection precision and accuracy in different scenarios. At the same time, fireworks detection can be performed in real time based on the pictures taken by the drone, which greatly improves the response speed.

[0042] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A schematic diagram of the system architecture of a smoke and fire detection system provided in an embodiment of the present application is shown.

[0045] Figure 2 A schematic diagram of the module architecture of an electronic device provided in an embodiment of the present application is shown.

[0046] Figure 3 One of the flow charts of the smoke and fire detection method provided in the embodiment of the present application is shown.

[0047] Figure 4 The second flowchart of the smoke and fire detection method provided in the embodiment of the present application is shown.

[0048] Figure 5 Shows Figure 4 Schematic diagram of the process of some sub-steps of step 21.

[0049] Figure 6 Shows Figure 4 Schematic diagram of the process of some sub-steps of step 25.

[0050] Figure 7 Shows Figure 3 Schematic diagram of the process flow of some sub-steps of step 13.

[0051] Figure 8 Shows Figure 7 A flowchart of some sub-steps of step 131 in FIG.

[0052] Fig. 9 Shows Figure 7 A flowchart of some sub-steps of step 133 in FIG.

[0053] Fig.10 Shows Figure 3 Schematic diagram of the process of some sub-steps of step 15.

[0054] Icons: 10-fireworks detection system; 110-training equipment; 120-client; 130-drone; 140-detection equipment; 150-early warning equipment; 20-electronic equipment; 210-memory; 220-processor; 230-communication module. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0056] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0057] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0058] The smoke and fire detection method provided in the embodiment of the present application can be applied to Figure 1 In the smoke and fire detection system 10 shown, referring to Figure 1 The smoke and fire detection system 10 includes a training device 110, a client 120, a drone 130, a detection device 140 and an early warning device 150. The detection device 140 can be communicated with the drone 130 and the early warning device 150 respectively through a network, a wireless connection, etc., and the training device 110 can be communicated with the detection device 140 through a wired or wireless method.

[0059] The user inputs user-specific data into the training device 110 via the client 120 .

[0060] The training device 110 is used to train a fire and smoke detection model using general data, and then adjust and train the fire and smoke detection model using user-specific data to obtain a personalized detection model, and then download and deploy the personalized detection model to the drone 130.

[0061] The drone 130 is used to collect images of the target area, obtain the images to be tested, and transmit the images to be tested to the detection device 140 in real time.

[0062] The detection device 140 is used to implement the smoke and fire detection method provided in the embodiment of the present application, including: inputting the image to be tested into the personalized detection model to obtain a segmented image output by the personalized detection model, the segmented image includes at least one segmented area and the category of the segmented area, and the personalized detection model is obtained by training the pre-trained smoke and fire detection model with user-specific data; performing contour detection and instance separation on the segmented area of ​​the segmented image to obtain a valid instance; in the case of a valid instance, according to the valid instance of the category of smoke and / or fire, issuing a smoke and fire alarm. Among them, the smoke and fire alarm can send warning information to the warning device 150.

[0063] The early warning device 150 is used to issue a fire alarm upon receiving early warning information.

[0064] The detection device 140 may be, but is not limited to: a cloud platform, an independent server, a server cluster, a personal computer, a laptop computer, etc. The warning device 150 may be, but is not limited to, a personal computer, a laptop computer, a mobile phone, a mobile terminal, an audible and visual alarm, or any warning device 150 that can receive information and issue an alarm.

[0065] Please refer to Figure 2 , is a block diagram of an electronic device 20, which may be Figure 1 The detection device 140 in the smoke and fire detection system 10 is shown. The electronic device 20 includes a memory 210, a processor 220 and a communication module 230. The memory 210, the processor 220 and the communication module 230 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0066] The memory 210 is used to store programs or data and can be, but not limited to, a random access memory, a read-only memory, a programmable read-only memory, an erasable read-only memory, an electrically erasable read-only memory, and the like.

[0067] The processor 220 is used to read / write data or programs stored in the memory 210 and execute corresponding functions. For example, Figure 1 In the smoke and fire detection system 10 shown, the processor 220 of the detection device 140 executes the computer program stored in the memory 210 to implement the smoke and fire detection method provided in the embodiment of the present application.

[0068] The communication module 230 is used to establish a communication connection between the electronic device 20 and other communication terminals through the network, and to send and receive data through the network. Figure 1 In the smoke and fire detection system 10 shown, the classmates module of the persistence device respectively sends and receives data to the drone 130 and the early warning device 150 through the network.

[0069] It should be understood that Figure 2 The structure shown is only a schematic diagram of the server structure. The electronic device 20 may also include Figure 2 More or fewer components as shown, or with Figure 2 Different configurations are shown. Figure 2 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0070] In order to solve the problem that the common fire and smoke detection has a slow response speed and the detection accuracy is difficult to meet the needs of different users, the embodiment of the present application provides a fire and smoke detection method, referring to Figure 3, including steps 11 to 15. And, Figure 1 The detection device 140 in the smoke and fire detection system 10 shown can Figure 2 The structure shown implements the execution of steps 11 to 15 when the processor 220 reads the computer program stored in the memory 210 .

[0071] Step 11: input the image to be tested into the personalized detection model to obtain a segmented image output by the personalized detection model.

[0072] The segmented image includes at least one segmented area and a category of the segmented area, and the personalized detection model is obtained by training a pre-trained fireworks detection model using user-specific data.

[0073] Step 13: Perform contour detection and instance separation on the segmented areas of the segmented image to obtain valid instances.

[0074] Step 15: If there is a valid instance, a fire alarm is issued according to the valid instance.

[0075] For example, in combination Figure 1 In the fire and smoke detection system 10 shown, the target user inputs user-specific data to the training device 110 through the client 120. The training device 110 uses the user-specific data to adjust and train the pre-trained fire and smoke detection model to obtain a personalized detection model for the target user, and downloads and deploys the personalized detection model to the detection device 140. The drone 130 regularly inspects the target area set by the target user according to the inspection cycle required by the target user, so as to collect images in real time, obtain the pictures to be tested, and transmit the pictures to be tested to the detection device 140 in real time.

[0076] The detection device 140 calls the personalized detection model of the target user, inputs the image to be tested into the personalized detection model, and obtains the segmented image output by the personalized detection model. Then, the detection device 140 performs contour detection and instance separation on the segmented area of ​​the segmented image to obtain a valid instance, and sends the fire and smoke warning information to the warning device 150 based on the valid instance if there is a valid instance. After receiving the fire and smoke warning information, the warning device 150 executes: displaying the warning information, initiating an audible and visual alarm, or any other alarm method.

[0077] In the above-mentioned fireworks detection method, in step 11 to step 15, a personalized detection model matching the user-specific data of the required scene is customized for the user, so that fireworks recognition is performed using the personalized detection model represented by the user's expected data, which greatly improves the detection precision and accuracy in different scenes. At the same time, fireworks detection can be performed in real time based on the pictures taken by the drone, which greatly improves the response speed.

[0078] There are many ways to obtain the personalized detection model used in step 11. For example, the model can be trained directly using the user's dedicated data, or the pre-trained model can be fine-tuned using the user's dedicated data. The implementation method is not limited.

[0079] In one example, in order to provide a model that reflects the user's expected detection precision and accuracy on the user's data and reduce the time spent on model training, the fireworks detection method provided in the embodiment of the present application introduces the concept of first trying to train a general fireworks detection model using general data, and then fine-tuning the fireworks detection model based on the dedicated data of each user to obtain a personalized detection model.

[0080] In this example, refer to Figure 4 , the embodiment of the present application obtains a personalized detection model through steps 21 to 25.

[0081] Step 21, using various common samples, iteratively train the initial model to obtain a fireworks detection model.

[0082] Step 23: Obtain a user-specific data set.

[0083] The user-specific data includes a plurality of special samples annotated by the user.

[0084] Step 25, using multiple dedicated samples, adjust and train the fireworks detection model to obtain a personalized detection model.

[0085] There are many ways to obtain the universal samples used in the above step 21. For example, you can capture fireworks pictures from network data, segment and annotate the fireworks pictures according to the areas of smoke and fire, and obtain universal samples. You can also filter the fireworks data on existing platforms and the Internet, select fireworks pictures from the perspective of drones, and filter a large number of fireworks pictures according to image indicators such as clarity. Then, segment and annotate each filtered fireworks picture according to the areas of smoke and fire in the picture to obtain the required number of universal samples. The implementation method is not limited.

[0086] There are also many ways to iteratively train the initial model to obtain a fireworks detection model. For example, you can use a general sample to iteratively train any YOLO series neural network to obtain a fireworks detection model. You can also use a general sample to iteratively train a Transformer-based model to obtain a fireworks detection model. The above methods are all examples, and their implementation methods are not limited.

[0087] In one example, in order to obtain a fireworks detection model with good recognition effect in more scenes, the SPD module and the pkinet module are added to the YOLOv11 neural network as the initial model in step 11, and the general samples are randomly cropped and resized, and the initial model is iteratively trained to obtain the concept of the fireworks detection model. Figure 5 In step 21, the following sub-steps 211 to 217 are used to iteratively train the initial model using various general samples to obtain a fireworks detection model.

[0088] Step 211 , segment and annotate the fireworks for each general image in the general data set to obtain general samples.

[0089] Step 213: randomly crop and resize each common sample to obtain a common sample set.

[0090] Step 215, adding the SPD module and the pkinet module to the YOLOv11 neural network to obtain an initial model.

[0091] Step 217, using the general sample set, iteratively train the initial model to obtain a fireworks detection model.

[0092] The SPD module is used to convert the spatial dimension of the input feature map into the depth dimension to enhance the network's learning of low-resolution images and small object targets. The pkinet module is used to perform multi-scale convolution and depth-separable convolution on the input feature map to enhance multi-scale changes at the network level, so that the YOLOv11 neural network can obtain features at multiple scales and perform feature integration.

[0093] In step 213, a universal sample is extracted from each universal sample by any preset random extraction method, and is cropped and / or resized to obtain a new universal sample, and the new universal sample and the previous universal sample are both put into the universal sample set.

[0094] In step 215, the SPD module and the pkinet module can be inserted into any position of the feature extraction layer of the YOLOv11 neural network, and the number of the SPD module and the pkinet module can be 1, 2, 2 and 3 respectively, and the specific positions of the two are not limited. In the iterative training process in step 217, the gradient descent method, the accelerated gradient method, the Newton method or any optimization algorithm can be used to optimize the model.

[0095] The above steps 211 to 217 obtain a general fireworks detection model that is adaptable to data of different scales and has good learning and reasoning capabilities for both high- and low-resolution images and small object targets, thereby ensuring the recognition effect of the fireworks detection model in various scenarios.

[0096] In other implementations, the YOLOv11 neural network used in the above steps 211 to 217 may be replaced by any other convolutional neural network.

[0097] After obtaining the fireworks detection model, there are multiple ways to adjust and train the fireworks detection model using multiple dedicated samples in step 25 to obtain a personalized detection model.

[0098] In one example, a plurality of dedicated samples and any optimization method are used to perform overall iterative training on the fire and smoke detection model to obtain a personalized detection model.

[0099] In another example, in order to meet the detection accuracy and speed required by users and improve the convergence speed of the model, in step 25, the idea of ​​only adjusting the number of layers of the segmentation head of the smoke and fire detection model according to customer needs, freezing the feature extraction layer in the early stage of training, and unfreezing the feature extraction layer in the later stage of training for joint fine-tuning is introduced. Figure 6 Step 25 uses multiple special samples to adjust and train the fireworks detection model, and the process of obtaining a personalized detection model includes steps 251 to 255.

[0100] Step 251, according to the user's index, the number of layers of the segmentation head of the fireworks detection model is adjusted to obtain a preliminary adjustment detection model.

[0101] Step 253, while freezing the feature extraction layer of the preliminary adjustment detection model, use the dedicated samples to iteratively train the preliminary adjustment detection model to obtain a preliminary result model.

[0102] Step 255, after unfreezing the feature extraction layer of the previous result model and lowering the learning rate, use the dedicated samples to iteratively train the previous result model to obtain a personalized detection model.

[0103] The user index may include any of several model performance indexes such as detection accuracy and detection speed. In step 251, the user index, the fire and smoke detection model and its current performance index may be sent to the terminal display screen of the technician, and the technician may adjust the number of layers of the segmentation head of the fire and smoke detection model to reduce or increase the number of layers to obtain the initial adjustment detection model.

[0104] Alternatively, an adjustment rule table may be pre-set, and the corresponding relationship between the difference of each indicator and the adjustment amount of the number of layers may be recorded in the adjustment rule table. Thus, in step 251, the training device calculates the difference between the detection accuracy required by the user and the detection accuracy of the fireworks detection model, queries the preset adjustment rule table based on the difference, obtains the adjustment number of layers, and makes adjustments based on the adjustment number of layers. The same is true for other indicators, which will not be described in detail here.

[0105] In step 253 and step 255, it can also be understood that step 253 is the early training and step 255 is the late training.

[0106] The conditions for ending the early iterative training can be set flexibly. For example, the number of model iterations can reach the set iteration threshold, the loss value of the model can reach the set loss threshold, or the loss value of the model can reach stability (i.e., model convergence). The above method is used as an example, and the conditions for ending the early iterative training are not limited.

[0107] Similar to the above conditions for ending the early iterative training, the conditions for ending the late iterative training can be when the number of model iterations reaches the set iteration threshold, when the model loss value reaches the set loss threshold, or when the model loss value reaches stability (i.e., model convergence), and the implementation method is not limited. In addition, during the late training process, the lowered learning rate is equal to the original learning rate minus the lowered value, and the lowered value can be 10 or 20, and its value is not limited.

[0108] During the early training and the later training, the optimization algorithm used can be gradient descent, method, accelerated gradient method, Newton's method or any other optimization algorithm, which is not limited here.

[0109] Through the above method, the fireworks detection model is fine-tuned into a personalized detection model based on user demand indicators. At the same time, the model is fine-tuned and trained using user-specific samples, thereby improving the precision, speed and accuracy of the final personalized detection model on user scenario data.

[0110] In step 11, the personalized detection model obtained in the above manner is used to obtain a segmented image of the image to be tested. The personalized detection model segments the image to be tested into a segmented image including multiple segmented areas (which can also be understood as a segmented mask image), and obtains the category of each segmented area, including smoke, fire, building, tree, etc.

[0111] Then, in step 13, contour detection and instance separation are performed on the segmented areas of the segmented image, and there are multiple ways to obtain valid instances.

[0112] In one example, contour detection may be directly performed on each segmented area to obtain a contour, and then a unique code may be assigned to each contour as an instance to obtain a valid example.

[0113] In another example, in order to reduce noise interference and improve the accuracy of valid examples, the idea of ​​first smoothing and filtering and then performing contour detection and instance separation is introduced in step 13. Figure 7In step 13, contour detection and instance separation are performed on the segmented areas of the segmented image, and the process of obtaining valid instances includes steps 131 to 135.

[0114] Step 131, smoothing the segmented area of ​​the segmented image classified as smoke or fire to obtain a smoothed image.

[0115] The smooth graph includes at least one smooth connected area.

[0116] Step 133, filtering each smooth connected area in the smooth image to obtain a corrected image.

[0117] The corrected image includes at least one valid area.

[0118] Step 135 , for each valid area, perform contour detection and instance separation on the valid area to obtain a valid instance.

[0119] In step 131, morphological operation, mean filtering, Gaussian filtering or any other smoothing method may be used, and the implementation method thereof is not limited.

[0120] In one example, in order to improve the accuracy of subsequent fire alarms and avoid repeated alarms for the same fire area, the idea of ​​performing an open operation and a kernel function processing on multiple partitions belonging to the same category is introduced in step 131. Figure 8 Step 131 smoothes the segmented area classified as smoke or fire in the segmented image, and the process of obtaining the smoothed image includes steps 1311 to 1313.

[0121] Step 1311 , segmented areas in the segmented image that belong to the same target category are grouped as the same category.

[0122] Step 1313, performing an opening operation on each segmented area of ​​the same category group in the segmented image, and then processing it using a kernel function of an elliptical kernel to obtain a smooth connected area.

[0123] Target categories include smoke and fire.

[0124] Both the opening operation and the kernel function are processing methods of morphological algorithms. The opening operation includes the process of corrosion followed by expansion, which can remove small objects, smooth boundaries and remove noise. At the same time, the adjacent (i.e., the correlation distance is extremely small and belongs to the same firework) segments are connected to merge into the same connected area. After the segments belonging to the same category group are processed by the opening operation, the adjacent segments are merged into a new segment. Then, the kernel function of the elliptical kernel smoothes and directs each segment after the opening operation to obtain the smooth connected area corresponding to each segment. In addition, the smooth connected area inherits the category of the original segment.

[0125] Through the above steps 1311 to 1313, the segmented areas belonging to the same smoke or fire are merged and connected, small objects and noise in the segmented areas are effectively removed, and smoothing is performed to make each smoothed connected area more accurate. In turn, it helps to avoid subsequent repeated alarms for the same smoke or fire area and improve the accuracy of subsequent smoke and fire alarms.

[0126] After the smoothed image is obtained in the above manner in step 131, the smoothed connected areas in the smoothed image are filtered in step 133, and the method of obtaining the corrected image can be flexibly set. For example, the smoothed image can be input into a pre-trained smoothing model, and the smoothing model is used to filter to obtain the corrected image. It can also be filtered according to preset rules. The implementation method is not limited.

[0127] In one example, refer to Fig. 9 Step 133 filters each smooth connected area in the smooth image, and the process of obtaining the corrected image includes steps 1331 to 1333.

[0128] Step 1331: for each smooth connected area, an area ratio is obtained according to the ratio of the area of ​​the smooth connected area to the total area of ​​the smoothed image.

[0129] Step 1333: filter out the smooth connected areas in the smooth image whose area ratio is less than the ratio threshold, and use the remaining smooth connected areas as valid areas to obtain a corrected image.

[0130] In the above process, the total area of ​​the smoothed image can be equal to the total number of pixels of the smoothed image, and the area of ​​the smoothed connected area can be equal to the total number of pixels of the smoothed connected area. The total area of ​​the smoothed image and the area of ​​the smoothed connected area can also be calculated using any area calculation method, and are not limited here.

[0131] The value of the ratio threshold can be 5% or 4%, and its value can vary with the area of ​​the real area corresponding to the image to be tested, without limitation. The ratio threshold can also be understood as a smoke and fire safety line or a false connection area. When the area ratio of the smooth connection area is less than the ratio threshold, it means that the smooth connection area is too small, and even smoke or fire will not cause a fire, or it is false or noise.

[0132] In the above manner, small smooth connected areas that are located below the fireworks safety line or that may be false or noisy are filtered to improve the detection precision and accuracy.

[0133] After filtering in the above manner, all unfiltered smooth connected areas are used as valid areas. Further, in step 135, for each valid area, contour detection is performed on the small area using Canny edge detection, boundary extraction, polygon fitting or any other contour detection method to obtain a contour. A unique code (such as color or ID, etc.) is assigned to each contour to obtain a valid instance, that is, a contour assigned with a unique code is a valid instance.

[0134] Furthermore, if there is a valid instance (classified as smoke or fire) after processing in the manner of steps 11 to 13, it can be determined that there is smoke or fire in the coverage area of ​​the image to be tested, which means that there may be a fire. In this case, in step 15, a warning message that there is a fire hazard in the coverage area can be generated and sent to the early warning device for timely alarm.

[0135] In one example, in order to make the alarm more accurate, the concept of making the alarm based on the location range of the valid instance is introduced in step 15. Fig.10 The process of issuing a fire alarm in step 15 includes steps 151 to 153.

[0136] Step 151, obtaining the location range of the valid instance according to the coverage area corresponding to the image to be tested.

[0137] Step 153, issuing an alarm message based on the location range.

[0138] The coordinates of the valid instance are converted between the pixel coordinate system, the camera coordinate system and the world coordinate system to obtain the position range of the valid instance.

[0139] Through the above method, the actual location information of the area where smoke or fire exists is obtained, and an alarm is issued accordingly to improve the accuracy of the alarm.

[0140] Based on the same concept as the smoke and fire detection method provided above, an embodiment of the present application further provides a smoke and fire detection device, including a detection module, a processing module and an alarm module.

[0141] The detection module is used to input the image to be tested into the personalized detection model to obtain a segmented image output by the personalized detection model. The segmented image includes at least one segmented area and a category of the segmented area. The personalized detection model is obtained by training the pre-trained fireworks detection model using user-specific data.

[0142] The processing module is used to perform contour detection and instance separation on the segmented areas of the segmented image to obtain valid instances.

[0143] The alarm module is used to issue a fire alarm according to the valid instance when there is a valid instance.

[0144] The above-mentioned fireworks detection device also includes a model training model, which is used to: use various general samples to iteratively train the initial model to obtain a fireworks detection model; obtain a user-specific data set, the user-specific data includes multiple special samples marked by the user; use multiple special samples to adjust and train the fireworks detection model to obtain a personalized detection model.

[0145] The above-mentioned fireworks detection device, under the coordinated action of the detection module, processing module and alarm module, customizes a personalized detection model that matches the user-specific data of the required scene for the user, so as to use the personalized detection model represented by the user's expected data to perform fireworks recognition, greatly improving the detection precision and accuracy in different scenes. At the same time, fireworks detection can be performed in real time based on the pictures taken by the drone, greatly improving the response speed.

[0146] For the specific implementation and effect of the smoke and fire detection device, please refer to the description of the implementation of the smoke and fire detection method above. For example, for the specific implementation and effect of the detection module, please refer to the description of the relevant content of step 11 above, for the specific implementation and effect of the processing module, please refer to the description of the relevant content of step 13 above, and for the specific implementation and effect of the alarm module, please refer to the description of the relevant content of step 15 above, which will not be repeated here.

[0147] In addition, each module of the above-mentioned smoke and fire detection device can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor 220 in the electronic device 20 in the form of hardware, or can be stored in the memory 210 of the electronic device 20 in the form of software, so that the processor 220 can call and execute the operations corresponding to the above-mentioned modules to implement the smoke and fire detection method provided above.

[0148] The embodiment of the present application also provides an electronic device 20, including a processor 220 and a memory 210, wherein the memory 210 stores a computer program executable by the processor 220, and the processor 220 can execute the computer program to implement the smoke and fire detection method provided above.

[0149] The embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by the processor 220, the smoke and fire detection method proposed in the embodiment of the present application is implemented.

[0150] In summary, the smoke and fire detection method, electronic device, and storage medium provided in the embodiments of the present application have at least the following beneficial effects:

[0151] (1) Use a user-customized personalized detection model to detect the presence of fireworks, improving the detection accuracy and speed on user data;

[0152] (2) It can be applied to the image processing stage of adaptive power inspection, which can effectively meet the needs of users and realize autonomous power inspection in the process of training models;

[0153] (3) Data scale transformation, SPD module and pkinet module are added to the personalized detection model to improve the detection effect of multi-scale and small target objects;

[0154] (4) It can be applied to subsequent fireworks recognition and any related thermal scenarios, solving the problem that users have little actual data but want to obtain better results.

[0155] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0156] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0157] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0158] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting fireworks, characterized in that: The method comprises: Inputting the image to be tested into the personalized detection model to obtain a segmented image output by the personalized detection model; wherein the segmented image includes at least one segmented area and a category of the segmented area, and the personalized detection model is obtained by training a pre-trained fireworks detection model using user-specific data; Performing contour detection and instance separation on the segmented areas of the segmented image to obtain valid instances; In the case that a valid instance exists, a smoke and fire alarm is issued according to the valid instance.

2. The method for detecting smoke and fire according to claim 1, characterized in that: The step of performing contour detection and instance separation on the segmented areas of the segmented image to obtain valid instances comprises: Smoothing the segmented areas of the category of smoke or fire in the segmented image to obtain a smoothed image; wherein the smoothed image includes at least one smooth connected area; Filtering each of the smooth connected areas in the smooth image to obtain a corrected image; wherein the corrected image includes at least one valid area; For each of the valid regions, contour detection and instance separation are performed on the valid region to obtain a valid instance.

3. The method for detecting smoke and fire according to claim 2, characterized in that: The step of filtering each of the smooth connected areas in the smooth image to obtain a corrected image includes: For each of the smooth connected areas, an area ratio is obtained according to a ratio of an area of ​​the smooth connected area to a total area of ​​the smooth image; The smooth connected areas whose area ratio in the smooth image is less than a ratio threshold are filtered out, and the remaining smooth connected areas are used as valid areas to obtain a corrected image.

4. The method for detecting smoke and fire according to claim 2, characterized in that: The step of smoothing the segmented area of ​​the category of smoke or fire in the segmented image to obtain a smoothed image comprises: The segmented areas in the segmented image that belong to the same target category are grouped as the same category; wherein the target category includes smoke and fire; An opening operation is performed on each segmented area of ​​the same category group in the segmented image, and then a kernel function of an elliptical kernel is used to process the segmented area to obtain a smooth connected area.

5. The method for detecting smoke and fire according to any one of claims 1 to 4, characterized in that: The step of obtaining the personalized detection model includes: Use various common samples to iteratively train the initial model to obtain a fireworks detection model; Acquire a user-specific data set; wherein the user-specific data includes a plurality of special samples annotated by the user; The fireworks detection model is adjusted and trained using the multiple dedicated samples to obtain a personalized detection model.

6. The method for detecting smoke and fire according to claim 5, characterized in that: The step of using the multiple dedicated samples to adjust and train the fireworks detection model to obtain a personalized detection model includes: According to the user index, the number of layers of the segmentation head of the fireworks detection model is adjusted to obtain a preliminary adjustment detection model; In the case of freezing the feature extraction layer of the preliminary adjustment detection model, using the special sample to iteratively train the preliminary adjustment detection model to obtain an early result model; After unfreezing the feature extraction layer of the previous result model and lowering the learning rate, the previous result model is iteratively trained using the dedicated samples to obtain a personalized detection model.

7. The method for detecting smoke and fire according to claim 5, characterized in that: The step of iteratively training the initial model using the general samples to obtain the smoke and fire detection model includes: Segment and annotate the fireworks for each general image in the general data set to obtain general samples; Randomly cropping and resizing each of the general samples to obtain a general sample set; An SPD module and a pkinet module are added to the YOLOv11 neural network to obtain an initial model; wherein the SPD module is used to convert the spatial dimension of the input feature map into a depth dimension, and the pkinet module is used to perform multi-scale convolution and depth-separable convolution on the input feature map; The initial model is iteratively trained using the universal sample set to obtain a smoke and fire detection model.

8. The method for detecting smoke and fire according to any one of claims 1 to 4, characterized in that: The step of issuing a fire alarm according to the effective example includes: Obtaining the location range of the valid instance according to the coverage area corresponding to the image to be tested; Based on the position range, an alarm message is issued.

9. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor can execute the computer program to implement the smoke and fire detection method according to any one of claims 1 to 8.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the smoke and fire detection method according to any one of claims 1 to 8 is implemented.

Images