A method and apparatus for detecting pyrotechnics

Abstract

The application provides a method for detecting fireworks: obtaining multiple frames of continuous images from video image data; performing motion detection on each frame of image to obtain a motion region of each frame of image; inputting the motion region into an image semantic segmentation neural network model to obtain a suspected smoke segmentation region and a suspected flame segmentation region of each frame of image; obtaining a target smoke segmentation region in the suspected smoke segmentation region and a target flame segmentation region in the suspected flame segmentation region; calculating a first region proportion ratio and a second region proportion ratio of the target smoke segmentation region and the target flame segmentation region in the motion region; if the first region proportion ratio and the second region proportion ratio of each of the images in a preset continuous frame of images adjacent to the frame of image are greater than or equal to a first preset threshold, then the target smoke segmentation region and the target flame segmentation region are respectively a real smoke region and a real flame region; if the real smoke region and the real flame region exist in each frame of image, and the real smoke region and the real flame region have an overlapping part, then a fireworks alarm is generated.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing and recognition, and particularly relates to a method and device for detecting fire. BACKGROUND

[0002] Fire is a natural disaster with a high occurrence probability, and how to effectively control the fire in the early stage to reduce property loss and even casualties is particularly important. In the prior art, smoke and fire detection is usually performed on images based on deep learning. If the detection result is a suspected smoke and fire detection area, a warning is directly given, and the detection result is not verified again. Therefore, there are still problems of serious false alarm and missed alarm, and the accuracy is low. It is necessary for a worker to confirm and verify whether a fire occurs at irregular time.

[0003] Therefore, the present application is provided. SUMMARY

[0004] The present application aims to overcome the shortcomings of the prior art and provide a method and device for detecting fire to solve the problem of low accuracy of smoke and fire detection in the prior art.

[0005] To solve the above technical problems, the basic idea of the technical solution of the present application is as follows:

[0006] In a first aspect, the present application provides a method for detecting fire, comprising:

[0007] obtaining a plurality of continuous images from video image data;

[0008] performing motion detection on each image to obtain a motion region of each image;

[0009] inputting the motion region into an image semantic segmentation neural network model to obtain a suspected smoke segmentation region and a suspected flame segmentation region of each image;

[0010] obtaining a target smoke segmentation region in the suspected smoke segmentation region and a target flame segmentation region in the suspected flame segmentation region, respectively; the target smoke segmentation region is a region surrounded by all first pixel points falling in the suspected smoke segmentation region and the motion region; and the target flame segmentation region is a region surrounded by all second pixel points falling in the suspected flame segmentation region and the motion region;

[0011] calculating a first region proportion ratio and a second region proportion ratio of the target smoke segmentation region and the target flame segmentation region in the motion region, respectively;

[0012] If the first area proportion ratio and the second area proportion ratio are both greater than or equal to a first preset threshold, it is determined that the target smoke segmentation region and the target flame segmentation region are real smoke region and real flame region respectively.

[0013] If the real smoke region and the real flame region exist in one frame of image, and the real smoke region and the real flame region have an overlapping part, a smoke and fire alarm is generated.

[0014] Optionally, the method further comprises:

[0015] If the real smoke region or the real flame region exists in a plurality of continuous frames of image, a smoke and fire alarm is generated.

[0016] Optionally, when the suspected smoke segmentation region is determined to be the suspected smoke segmentation region in the frame of image, the motion region is input to an image semantic segmentation neural network model for recognition to obtain the suspected smoke segmentation region and the suspected flame segmentation region of each frame of image, comprising:

[0017] For the motion region of each frame of image, each pixel point of the motion region is identified by using the image semantic segmentation neural network model to obtain the probability value of each pixel point being light smoke, thick smoke, small fire or large fire respectively;

[0018] The type of each pixel point is determined according to the maximum probability value of each pixel point;

[0019] The pixel points of the light smoke type and the thick smoke type are integrated to obtain the suspected smoke segmentation region;

[0020] The pixel points of the small fire type and the large fire type are integrated to obtain the suspected flame segmentation region.

[0021] Optionally, the suspected smoke segmentation region is obtained according to three adjacent frames of image, comprising:

[0022] For each frame of image, the suspected smoke segmentation region of the frame of image is obtained by merging the suspected smoke segmentation region of the frame of image and the suspected smoke segmentation region of two adjacent frames of image respectively;

[0023] For each frame of image, the suspected flame segmentation region of the frame of image is obtained by merging the suspected flame segmentation region of the frame of image and the suspected flame segmentation region of two adjacent frames of image respectively.

[0024] Optionally, the target smoke segmentation region in the suspected smoke segmentation region and the target flame segmentation region in the suspected flame segmentation region are obtained respectively, comprising:

[0025] respectively acquire coordinate values of each pixel point contained in each of the suspected smoke segmentation area, the suspected flame segmentation area and the motion area;

[0026] take the pixel points with the same coordinate values in the suspected smoke segmentation area and the motion area as the first pixel points, and generate the target smoke segmentation area according to all the first pixel points;

[0027] take the pixel points with the same coordinate values in the suspected flame segmentation area and the motion area as the second pixel points, and generate the target flame segmentation area according to all the second pixel points.

[0028] Optionally, the separately calculating the first area ratio ratio and the second area ratio ratio of the target smoke segmentation area and the target flame segmentation area in the motion area comprises:

[0029] taking the ratio of the number of all the first pixel points contained in the target smoke segmentation area to the total number of all pixel points in the motion area as the first area ratio ratio;

[0030] taking the ratio of the number of all the second pixel points contained in the target flame segmentation area to the total number of all pixel points in the motion area as the second area ratio ratio.

[0031] In a second aspect, the present application provides a device for detecting fireworks, comprising:

[0032] a first acquisition module, configured to acquire a plurality of continuous images from video image data;

[0033] a motion detection module, configured to perform motion detection on each image to obtain a motion area of each image;

[0034] an identification module, configured to input the motion area into an image semantic segmentation neural network model to obtain a suspected smoke segmentation area and a suspected flame segmentation area of each image;

[0035] a second acquisition module, configured to separately acquire a target smoke segmentation area in the suspected smoke segmentation area and a target flame segmentation area in the suspected flame segmentation area; the target smoke segmentation area is an area surrounded by all first pixel points falling in the suspected smoke segmentation area and the motion area at the same time; and the target flame segmentation area is an area surrounded by all second pixel points falling in the suspected flame segmentation area and the motion area at the same time;

[0036] a calculation module, configured to separately calculate a first area ratio ratio and a second area ratio ratio of the target smoke segmentation area and the target flame segmentation area in the motion area.

[0037] The first determining module is configured to determine that the target smoke segmentation region and the target flame segmentation region are a real smoke region and a real flame region respectively when the first area proportion ratio and the second area proportion ratio are greater than or equal to a first preset threshold.

[0038] The second determining module is configured to generate a smoke and fire alarm if the real smoke region and the real flame region exist in one frame of image and there is an overlapping part between the real smoke region and the real flame region.

[0039] Optionally, the second determining module further includes:

[0040] The first determining unit is configured to generate a smoke and fire alarm if the real smoke region or the real flame region exists in a plurality of continuous frames of image.

[0041] Optionally, the identifying module includes:

[0042] The first identifying unit is configured to identify each pixel point of a motion region of each frame of image by using an image semantic segmentation neural network model to obtain a probability value of each pixel point being light smoke, heavy smoke, small fire or large fire.

[0043] The second determining unit is configured to determine the type of each pixel point according to a maximum probability value of each pixel point.

[0044] The first integrating unit is configured to integrate the pixel points of the light smoke type and the heavy smoke type to obtain the suspected smoke segmentation region.

[0045] The second integrating unit is configured to integrate the pixel points of the small fire type and the large fire type to obtain the suspected flame segmentation region.

[0046] Optionally, the identifying module further includes:

[0047] The third integrating unit is configured to combine the suspected smoke segmentation region of each of a frame of image and two frames of image adjacent to the frame of image to obtain the suspected smoke segmentation region of the frame of image.

[0048] The fourth integrating unit is configured to combine the suspected flame segmentation region of each of a frame of image and two frames of image adjacent to the frame of image to obtain the suspected flame segmentation region of the frame of image.

[0049] Optionally, the second acquiring module further includes:

[0050] The first acquisition unit is used for acquiring coordinate values of each pixel point contained in each of the suspected smoke segmentation area, the suspected flame segmentation area and the motion area respectively.

[0051] The first generation unit is used for taking the pixel points with the same coordinate values in the suspected smoke segmentation area and the motion area as the first pixel points, and generating the target smoke segmentation area according to all the first pixel points.

[0052] The second generation unit is used for taking the pixel points with the same coordinate values in the suspected flame segmentation area and the motion area as the second pixel points, and generating the target flame segmentation area according to all the second pixel points.

[0053] Optionally, the calculation module further comprises:

[0054] The first calculation unit is used for taking a ratio of the number of all the first pixel points contained in the target smoke segmentation area to the total number of all the pixel points in the motion area as the first area ratio value.

[0055] The second calculation unit is used for taking a ratio of the number of all the second pixel points contained in the target flame segmentation area to the total number of all the pixel points in the motion area as the second area ratio value.

[0056] The present application provides a kind of method for detecting fireworks, comprising: first, obtain multiple frames of continuous images from video image data;Then, each frame of image is carried out motion detection and obtains the motion area of each frame of image;Second, the motion area is input to image semantic segmentation neural network model and is identified to obtain the suspected smoke segmentation area and suspected flame segmentation area of each frame of image;Respectively, the target smoke segmentation area in the suspected smoke segmentation area and the target flame segmentation area in the suspected flame segmentation area are obtained;The target smoke segmentation area is the region surrounded by all first pixel points in the suspected smoke segmentation area and the motion area;The target flame segmentation area is the region surrounded by all second pixel points in the suspected flame segmentation area and the motion area;The first area ratio value and the second area ratio value of the target smoke segmentation area and the target flame segmentation area in the motion area are calculated respectively;If the first area ratio value and the second area ratio value are greater than or equal to first preset threshold, then the target smoke segmentation area and the target flame segmentation area are real smoke area and real flame area respectively;Finally, if the real smoke area and the real flame area exist in a frame of image, and the real smoke area and the real flame area exist overlap part, then generate fireworks alarm.

[0057] The present application firstly processes the image to obtain a motion region of the image, and then processes the motion region to obtain a suspected smoke segmentation region and a suspected flame segmentation region, and further obtains a target smoke segmentation region and a target flame segmentation region, determines whether there is a real smoke region and a real flame region through verification of the target smoke segmentation region and the target flame segmentation region, and finally generates a smoke and flame alarm. The present application processes the image in stages, not only reduces the data processing amount and reduces the influence of background light interference factors, but also excludes false detection through secondary verification of the detected region, and further improves the detection accuracy.

[0058] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0059] The accompanying drawings, which are part of the present application, serve to provide a further understanding of the present application, the illustrative embodiments of the present application and their descriptions serve to explain the present application, but do not constitute an improper limitation on the present application. Obviously, the drawings described below are only some embodiments, and other drawings can be obtained by those skilled in the art without creative labor. In the drawings:

[0060] Fig. 1 is a flowchart of a method for detecting smoke and fire provided by an embodiment of the present application;

[0061] Fig. 2 is a schematic diagram of a device for detecting smoke and fire provided by an embodiment of the present application.

[0062] It should be noted that these drawings and written descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments will be described clearly and completely below with reference to the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but not to limit the scope of the present application.

[0064] In the description of the present application, it should be noted that the terms "upper", "lower", "front", "rear", "left", "right", "vertical", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore it cannot be understood as a limitation on the present application.

[0065] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood in a broad sense, for example, can be fixed connection, can also be detachable connection, or integrally connected; can be mechanical connection, can also be electrical connection; can be directly connected, can also be indirectly connected through an intermediate medium. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0066] As Figs. 1-2 shown, the method for detecting fireworks provided by the embodiment of the present application comprises:

[0067] S101, obtaining multiple continuous images from video image data;

[0068] S102, performing motion detection on each image to obtain a motion region of each image;

[0069] S103, inputting the motion region into an image semantic segmentation neural network model to identify a suspected smoke segmentation region and a suspected flame segmentation region of each image;

[0070] S104, obtaining a target smoke segmentation region in the suspected smoke segmentation region and a target flame segmentation region in the suspected flame segmentation region, respectively; the target smoke segmentation region is a region surrounded by all first pixel points falling in the suspected smoke segmentation region and the motion region; the target flame segmentation region is a region surrounded by all second pixel points falling in the suspected flame segmentation region and the motion region;

[0071] S105, calculating a first region proportion ratio and a second region proportion ratio of the target smoke segmentation region and the target flame segmentation region in the motion region, respectively;

[0072] S106, if the first region proportion ratio and the second region proportion ratio are both greater than or equal to a first preset threshold, then it is determined that the target smoke segmentation region and the target flame segmentation region are real smoke region and real flame region, respectively;

[0073] S107, if the real smoke region and the real flame region exist in one image, and there is an overlapping part between the real smoke region and the real flame region, then generating a fireworks alarm.

[0074] In the above step S101, the video image data is a monitoring video captured by a camera.

[0075] Specifically, the server obtains a monitoring video of a certain scene collected by a camera in real time, and obtains multiple continuous images after processing.

[0076] In step S102, the motion region is a region in which pixel points dynamically change, and the motion region can include fireworks, moving vehicles, moving organisms, and the like.

[0077] Specifically, the server performs motion detection on each frame of image according to the frame of image and two adjacent frames of image of the frame of image, and further obtains the motion region of the frame of image.

[0078] For example, taking the motion detection on the nth frame of image to obtain the motion region of the nth frame of image as an example, first, the (n-1)th frame of image is obtained, the pixel value of each pixel point in the nth frame of image is compared with the pixel value of each pixel point in the (n-1)th frame of image, the pixel points with changed pixel values are retained, and the region in which the pixel points with changed pixel values are located is intercepted as the difference image d n-1 between the nth frame of image and the (n-1)th frame of image. Similarly, the (n+1)th frame of image is obtained, the pixel value of each pixel point in the nth frame of image is compared with the pixel value of each pixel point in the (n+1)th frame of image, the pixel points with changed pixel values are retained, and the region in which the pixel points with changed pixel values are located is intercepted as the difference image d n between the nth frame of image and the (n+1)th frame of image.

[0079] Further, the pixel points with pixel values less than a second preset threshold in the frame difference images d n-1 and d n are all set to 0, the pixel points with pixel values greater than or equal to the second preset threshold are all set to 255, and the regions in which the pixel points with pixel value 255 are located in the two frame difference images are merged to generate the motion region of the nth frame of image.

[0080] Since the reference size input to each image semantic segmentation neural network model is different, when the to-be-segmented region of each frame of image is cropped, the image size input to the image semantic segmentation neural network model to be used needs to be referred to. In order to input the motion region of each frame of image in step 2 to each image semantic segmentation neural network model in a suitable size, the motion region is input to the image semantic segmentation neural network model to obtain the suspected smoke segmentation region and the suspected flame segmentation region of each frame of image, and step 103 further includes:

[0081] The image size W and H of the image semantic segmentation neural network model to which the motion region is input are obtained, and the diagonal line coordinates (minX, minY), (maxX, maxY) of the circumscribed rectangle in which the motion region is located are obtained. If W < maxX-minX, the X axis is intercepted according to minX and maxX of the circumscribed rectangle, if W ≥ maxX-minX, the X axis is intercepted according to and .

[0082] If H < maxY-minY, the Y axis is intercepted according to minY and maxY of the positive circumscribed rectangle, and if H ≥ maxY-minY, the Y axis is intercepted according to and .

[0083] In the above step S103, the suspected smoke segmentation region contains a real smoke region and a smoke region part mistakenly considered as smoke due to existing interference factors in the image. The suspected flame segmentation region contains a real fire region and a fire region part mistakenly considered as fire due to existing interference factors in the image.

[0084] Specifically, the motion region of each frame of image is input into the image semantic segmentation neural network model for identification to obtain the suspected smoke segmentation region and the suspected flame segmentation region of the frame of image.

[0085] However, the actual suspected smoke / flame segmentation region of a frame of image may be the suspected smoke / flame segmentation region in the frame of image itself, or may be obtained according to the suspected smoke / flame segmentation regions of three adjacent images respectively.

[0086] When it is judged that the suspected smoke segmentation region is the suspected smoke segmentation region in the frame of image, and the suspected flame segmentation region is the suspected flame segmentation region in the frame of image, the motion region of each frame of image is input into the image semantic segmentation neural network model for identification to obtain the suspected smoke segmentation region and the suspected flame segmentation region of each frame of image, S103 comprising:

[0087] S1031, for the motion region of each frame of image, the image semantic segmentation neural network model is used to identify each pixel point of the motion region to obtain the probability value of each pixel point belonging to light smoke, heavy smoke, small fire and large fire respectively;

[0088] S1032, determining the type of each pixel point according to the maximum probability value of each pixel point;

[0089] S1033, integrating the pixel points of light smoke type and heavy smoke type to obtain the suspected smoke segmentation region;

[0090] S1034, integrating the pixel points of small fire type and large fire type to obtain the suspected flame segmentation region.

[0091] In the above step S1031, specifically, the motion region of each frame of image is input into the image semantic segmentation neural network model to respectively identify each pixel point for light smoke identification, heavy smoke identification, small fire identification and large fire identification, and further obtain the probability value of each pixel point belonging to light smoke, heavy smoke, small fire and large fire respectively.

[0092] For example, inputting the motion region of each frame of image into the image semantic segmentation neural network model, different colors such as red, yellow, blue and the like are obtained, and red represents the color of the same type such as smoke, yellow may represent the color of a living being, and blue may represent the color of a background, and different color regions are composed of different pixel points. In the present application, a SegNet network is used to identify the motion region of each frame of image, and the network mainly includes an Encoder, a Decoder and a soft-max classifier. The Encoder process is a convolution operation on the input source image to extract features, and the receptive field is increased through a pooling layer while the image is gradually reduced. The Decoder process is to reproduce the features of the classified image through deconvolution, restore the original size of the image through upsampling, and finally output the maximum value of different classifications through the soft-max classifier to obtain the final segmentation map.

[0093] Taking an A pixel point in the motion region of a frame of image as an example, the image semantic segmentation neural network model is used to identify the probability values of the A pixel point belonging to light smoke, thick smoke, small fire and large fire respectively, and the four probability values are a1, a2, a3 and a4 in turn. Secondly, the image semantic segmentation neural network model can also be used to identify the probability value of the A pixel point belonging to a living being / background.

[0094] In the above step S1032, specifically, the four probability values of each pixel point are compared, and the type of each pixel point is determined according to the type corresponding to the maximum probability value.

[0095] For example, if the maximum of the above four probability values is a 2， And a2 represents the probability of the pixel point belonging to thick smoke, so it is determined that the pixel point belongs to the thick smoke type.

[0096] In the above step S1033, specifically, the region surrounded by the pixel points of the light smoke type and the pixel points of the thick smoke type is taken as the suspected smoke segmentation region in the motion region, and the region surrounded by the pixel points of the light smoke type and the pixel points of the thick smoke type is segmented from the motion region to obtain the suspected smoke segmentation region.

[0097] In the above step S1034, specifically, the region surrounded by the pixel points of the small fire type and the pixel points of the large fire type is taken as the suspected flame segmentation region in the motion region, and the region surrounded by the pixel points of the small fire type and the pixel points of the large fire type is segmented from the motion region to obtain the suspected flame segmentation region. Similarly, the suspected living being segmentation region is also obtained in the same way.

[0098] The suspected smoke segmentation region and the suspected flame segmentation region are obtained according to three adjacent frames of image, and specifically include:

[0099] S1035, for each frame image, merging the suspected smoke segmentation region of the frame image and the suspected smoke segmentation region of each of the two frame images adjacent to the frame image to obtain the suspected smoke segmentation region of the frame image;

[0100] S1036, for each frame image, merging the suspected flame segmentation region of the frame image and the suspected flame segmentation region of each of the two frame images adjacent to the frame image to obtain the suspected flame segmentation region of the frame image.

[0101] In the step 1035, specifically, the two frame images adjacent to each frame image are determined according to the frame image, and the suspected smoke segmentation regions of the above-mentioned three adjacent frame images are merged to obtain the actual suspected smoke segmentation region of the frame image.

[0102] For example, taking the nth frame image as an example, the two frame images adjacent to the nth frame image are determined as the (n-1)th frame and the (n+1)th frame, and the suspected smoke segmentation regions of the (n-1)th frame, the nth frame and the (n+1)th frame are respectively region 1, region 2 and region 3, the above-mentioned three regions are merged and the overlapping region is removed, and the finally merged region is taken as the actual suspected smoke segmentation region of the nth frame image. The present application also provides conditions under which the suspected segmentation region of a frame image is obtained according to three adjacent frame images. Taking the suspected smoke segmentation region of the nth frame image as an example, there are n pixel points in the nth frame image, k pixel points in the (n-1)th frame image, the number of pixel points with the same coordinates in the nth frame image and the (n-1)th frame image is x, the ratio of x and n+k-x is calculated, and it is judged whether the ratio is greater than or equal to the second preset threshold value. Similarly, there are n pixel points in the nth frame image, m pixel points in the (n+1)th frame image, the number of pixel points with the same coordinates in the nth frame image and the (n+1)th frame image is y, the ratio of y and n+m-y is calculated, and it is judged whether the ratio is greater than or equal to the second preset threshold value. If the above-mentioned two ratios are greater than or equal to the second preset threshold value, the suspected smoke segmentation regions of the (n-1)th frame, the nth frame and the (n+1)th frame are merged and the repeated region is removed, and then the suspected smoke segmentation region of the nth frame image is obtained.

[0103] In the step 1036, specifically, the two frame images adjacent to each frame image are determined according to the frame image, and the suspected flame segmentation regions of the above-mentioned three adjacent frame images are merged to obtain the actual suspected flame segmentation region of the frame image.

[0104] In the step 104, the target smoke segmentation region is a segmentation region containing only smoke, and the target flame segmentation region is a segmentation region containing only flame.

[0105] Specifically, the target smoke segmentation region in the suspected smoke segmentation region of each frame of image and the target flame segmentation region in the suspected flame segmentation region of the frame of image are respectively acquired.

[0106] To understand the step S104 in more detail, it further includes:

[0107] S1041, the coordinate values of each pixel point contained in the suspected smoke segmentation region, the suspected flame segmentation region and the motion region are respectively acquired.

[0108] S1042, the pixel points with the same coordinate values in the suspected smoke segmentation region and the motion region are taken as the first pixel points, and the target smoke segmentation region is generated according to all the first pixel points.

[0109] S1043, the pixel points with the same coordinate values in the suspected flame segmentation region and the motion region are taken as the second pixel points, and the target flame segmentation region is generated according to all the second pixel points.

[0110] In the step S041, for a frame of image, the coordinate values of each pixel point in the suspected smoke segmentation region of the frame of image, the coordinate values of each pixel point in the suspected flame segmentation region of the frame of image and the coordinate values of each pixel point in the motion region of the frame of image are acquired.

[0111] For example, when there are two pixel points in the suspected smoke segmentation region, the coordinate values of the two pixel points are respectively acquired. Similarly, the coordinate values of the pixel points in the suspected flame segmentation region and the motion region are acquired in the same way.

[0112] In the step S1042, the coordinate values are the values of the pixel points on the horizontal coordinate and the values of the pixel points on the vertical coordinate.

[0113] The horizontal and vertical coordinate values of each pixel point in the suspected smoke segmentation region are compared with the horizontal and vertical coordinate values of each pixel point in the motion region, a plurality of pixel points with the same horizontal and vertical coordinate values are obtained, the pixel points are taken as the first pixel points, and the region surrounded by all the first pixel points is taken as the target suspected smoke segmentation region.

[0114] For example, the n-th frame image suspected smoke segmentation region has 3 pixel points, and the coordinate values of the three pixel points are (x1, y1), (x2, y2), and (x3, y3). The n-th frame image motion region has 4 pixel points, and the coordinate values are (u1, v1), (u2, v2), (u3, v3), and (u4, v4). Compare (x1, y1), (x2, y2), (x3, y3) with (u1, v1), (u2, v2), (u3, v3), and (u4, v4), respectively. If x1 is equal to u2 and y1 is equal to y1, then the pixel point corresponding to (x1, y1) and the pixel point corresponding to (u2, v2) are regarded as the same pixel point, that is, the first pixel point. If there are 2 first pixel points, then the region surrounded by the 2 first pixel points is regarded as the target suspected smoke segmentation region.

[0115] In the above step S1043, the horizontal and vertical coordinate values of each pixel point in the above suspected flame segmentation region are compared with the horizontal and vertical coordinate values of each pixel point in the motion region, and a plurality of pixel points with the same horizontal and vertical coordinate values are obtained. These pixel points are regarded as second pixel points, and the region surrounded by all the second pixel points is regarded as the target suspected smoke segmentation region.

[0116] For example, the n-th frame image suspected flame segmentation region has 4 pixel points, and the coordinate values of the three pixel points are (m1, n1), (m2, n2), (m3, n3), and (m4, n4). The n-th frame image motion region has 3 pixel points, and the coordinate values are (j1, k1), (j2, k2), and (j3, k3). Compare (m1, n1), (m2, n2), (m3, n3), and (m4, n4) with (j1, k1), (j2, k2), and (j3, k3), respectively. If m3 is equal to j1 and n3 is equal to k1, then the pixel point corresponding to (m3, n3) and the pixel point corresponding to (j1, k1) are regarded as the same pixel point, that is, the second pixel point. If there are 3 second pixel points, then the region surrounded by the 3 second pixel points is regarded as the target suspected flame segmentation region.

[0117] In the above step 105, for each frame image, the first proportion of the target smoke segmentation region in the motion region of the frame image is calculated, that is, the first area proportion ratio; and the second proportion of the target flame segmentation region in the motion region of the frame image is calculated, that is, the second area proportion ratio.

[0118] In order to more clearly understand the calculation of the first area proportion ratio and the second area proportion ratio of the target smoke segmentation region and the target flame segmentation region in the motion region, respectively, step S105 includes:

[0119] S1051, taking the ratio of the number of all the first pixel points contained in the target smoke segmentation region to the total number of all pixel points in the motion region as the first region proportion ratio;

[0120] S1052, taking the ratio of the number of all the second pixel points contained in the target flame segmentation region to the total number of all pixel points in the motion region as the second region proportion ratio.

[0121] In the above step S1051, the number of all the first pixel points in the target smoke segmentation region is p, and the total number of all the pixel points in the motion region is q, and p / q is taken as the first region proportion ratio.

[0122] In the above step S1052, the number of all the second pixel points in the target flame segmentation region is r, and the total number of all the pixel points in the motion region is q, and r / q is taken as the second region proportion ratio.

[0123] In the above step S106, it is determined whether the first region proportion ratio is greater than or equal to a first preset threshold value, and whether the second region proportion ratio is greater than or equal to the first preset threshold value. If the first region proportion ratio is greater than or equal to the first preset threshold value, it is determined that the target smoke segmentation region is a real smoke segmentation region; if the second region proportion ratio is greater than or equal to the first preset threshold value, it is determined that the target flame segmentation region is a real flame segmentation region.

[0124] In the above step S107, if the real smoke segmentation region and the real flame segmentation region exist in one frame of image, and the real smoke segmentation region and the real flame segmentation region overlap in the horizontal coordinate or overlap in the vertical coordinate, a fireworks alarm is directly generated.

[0125] In addition to the above step S107, the method for determining fireworks further comprises:

[0126] S108, if the real smoke region or the real flame region exists in a plurality of continuous frames of images, a fireworks alarm is generated.

[0127] If only the real smoke region exists in a plurality of continuous frames of images, a fireworks alarm can also be generated; if only the real flame region exists in a plurality of continuous frames of images, a fireworks alarm can also be generated. The plurality of continuous frames can be more than 3 frames, which is not limited in the present application.

[0128] ​​In a second aspect, the present application provides a device for detecting fireworks, comprising: a first acquisition module 201, a motion detection module 202, an identification module 203, a second acquisition module 204, a calculation module 205, a first determination module 206, and a second determination module 207.

[0129] The first acquisition module 201 is configured to acquire a plurality of continuous images from video image data.

[0130] The motion detection module 202 is configured to perform motion detection on each image to obtain a motion region of each image.

[0131] The identification module 203 is configured to input the motion region into an image semantic segmentation neural network model for identification to obtain a suspected smoke segmentation region and a suspected flame segmentation region of each image.

[0132] The second acquisition module 204 is configured to acquire a target smoke segmentation region in the suspected smoke segmentation region and a target flame segmentation region in the suspected flame segmentation region, respectively; the target smoke segmentation region is a region surrounded by all first pixel points that fall in both the suspected smoke segmentation region and the motion region; and the target flame segmentation region is a region surrounded by all second pixel points that fall in both the suspected flame segmentation region and the motion region.

[0133] The calculation module 205 is configured to calculate a first region proportion ratio and a second region proportion ratio of the target smoke segmentation region and the target flame segmentation region in the motion region, respectively.

[0134] The first determination module 206 is configured to determine that the target smoke segmentation region and the target flame segmentation region are real smoke regions and real flame regions, respectively, when both the first region proportion ratio and the second region proportion ratio are greater than or equal to a first preset threshold.

[0135] The second determination module 207 is configured to generate a fireworks alarm if both the real smoke region and the real flame region exist in a frame of image, and the real smoke region and the real flame region have an overlapping part.

[0136] Optionally, the second determination module further comprises:

[0137] A first determination unit is configured to generate a fireworks alarm if both the real smoke region and the real flame region exist in a plurality of continuous images.

[0138] Optionally, the identification module comprises:

[0139] The first identification unit is configured to identify each pixel point of the motion region of each frame of image by using an image semantic segmentation neural network model to obtain probability values of each pixel point being light smoke, thick smoke, small fire or large fire respectively.

[0140] The second determination unit is configured to determine the type of each pixel point according to the maximum probability value of each pixel point.

[0141] The first integration unit is configured to integrate the pixel points of the light smoke type and the thick smoke type to obtain the suspected smoke segmentation region.

[0142] The second integration unit is configured to integrate the pixel points of the small fire type and the large fire type to obtain the suspected flame segmentation region.

[0143] Optionally, the identification module further comprises:

[0144] The third integration unit is configured to combine the suspected smoke segmentation regions of each of the frame of image and two adjacent frames of image of the frame of image to obtain the suspected smoke segmentation region of the frame of image.

[0145] The fourth integration unit is configured to combine the suspected flame segmentation regions of each of the frame of image and two adjacent frames of image of the frame of image to obtain the suspected flame segmentation region of the frame of image.

[0146] Optionally, the second acquisition module further comprises:

[0147] The first acquisition unit is configured to acquire coordinate values of each pixel point included in the suspected smoke segmentation region, the suspected flame segmentation region and the motion region respectively.

[0148] The first generation unit is configured to take the pixel points with the same coordinate values in the suspected smoke segmentation region and the motion region as the first pixel points, and generate the target smoke segmentation region according to all the first pixel points.

[0149] The second generation unit is configured to take the pixel points with the same coordinate values in the suspected flame segmentation region and the motion region as the second pixel points, and generate the target flame segmentation region according to all the second pixel points.

[0150] Optionally, the calculation module further comprises:

[0151] The first calculation unit is configured to take the ratio of the number of all the first pixel points included in the target smoke segmentation region to the total number of all the pixel points in the motion region as the first area proportion ratio.

[0152] A second calculation unit is configured to take a ratio of a total number of all second pixel points contained in the target flame segmentation region to a total number of all pixel points in the motion region as the second region proportion ratio.

[0153] The above merely describes preferred embodiments of the present application and is not intended to limit the present application in any form. Although the present application has been described above with reference to preferred embodiments, the present application is not intended to be limited to the above embodiments. Any person skilled in the art can make some changes or modifications to the above-mentioned technical content without departing from the technical solution of the present application, and equivalent embodiments with equivalent changes are equivalent. The embodiments in the above examples can be further combined or replaced, but as long as it does not deviate from the technical solution of the present application, any simple modification, equivalent change and modification of the above examples according to the technical essence of the present application are still within the scope of the present application.

Claims

1. A method of detecting pyrotechnic material, characterized by: The method comprises the following steps: obtaining a plurality of continuous images from video image data; performing motion detection on each image to obtain a motion region of each image; inputting the motion region into an image semantic segmentation neural network model to obtain a suspected smoke segmentation region and a suspected flame segmentation region of each image; obtaining a target smoke segmentation region in the suspected smoke segmentation region and a target flame segmentation region in the suspected flame segmentation region; the target smoke segmentation region is a region surrounded by all first pixel points falling in the suspected smoke segmentation region and the motion region; the target flame segmentation region is a region surrounded by all second pixel points falling in the suspected flame segmentation region and the motion region; calculating a first region proportion ratio and a second region proportion ratio of the target smoke segmentation region and the target flame segmentation region in the motion region; if the first region proportion ratio and the second region proportion ratio are both greater than or equal to a first preset threshold, then the target smoke segmentation region and the target flame segmentation region are real smoke region and real flame region respectively; if the real smoke region and the real flame region exist in one image, and there is an overlapping part between the real smoke region and the real flame region, then generating a smoke and fire alarm; the method for obtaining the target smoke segmentation region in the suspected smoke segmentation region and the target flame segmentation region in the suspected flame segmentation region comprises the following steps: obtaining coordinate values of each pixel point included in the suspected smoke segmentation region, the suspected flame segmentation region and the motion region; taking the pixel points with the same coordinate values in the suspected smoke segmentation region and the motion region as the first pixel points, and generating the target smoke segmentation region according to all the first pixel points; taking the pixel points with the same coordinate values in the suspected flame segmentation region and the motion region as the second pixel points, and generating the target flame segmentation region according to all the second pixel points.

2. A method of detecting a pyrotechnic charge according to claim 1, wherein The method further comprises the following steps: if the real smoke region or the real flame region exists in a plurality of continuous images, then generating a smoke and fire alarm.

3. A method of detecting a pyrotechnic charge as claimed in claim 1, wherein when the suspected smoke segmentation region is a suspected smoke segmentation region in the image and the suspected flame segmentation region is a suspected flame segmentation region in the image, inputting the motion region into the image semantic segmentation neural network model to obtain the suspected smoke segmentation region and the suspected flame segmentation region of each image, comprising the following steps: for the motion region of each image, identifying each pixel point of the motion region by using the image semantic segmentation neural network model to obtain probability values of each pixel point being light smoke, thick smoke, small fire and large fire respectively; determining the type of each pixel point according to the maximum probability value of each pixel point; integrating the pixel points of the light smoke type and the thick smoke type to obtain the suspected smoke segmentation region; integrating the pixel points of the small fire type and the large fire type to obtain the suspected flame segmentation region.

4. A method of detecting a pyrotechnic charge according to claim 3, wherein, the suspected smoke segmentation region and the suspected flame segmentation region are obtained according to three adjacent images, comprising the following steps: merge the suspected smoke segmentation region of the frame image and the suspected smoke segmentation region of each of two images adjacent to the frame image to obtain the suspected smoke segmentation region of the frame image; merge the suspected flame segmentation region of the frame image and the suspected flame segmentation region of each of two images adjacent to the frame image to obtain the suspected flame segmentation region of the frame image.

5. The method of claim 1, wherein the method further comprises: the first area ratio ratio and the second area ratio ratio of the target smoke segmentation region and the target flame segmentation region in the motion region are calculated respectively, including: the first area ratio ratio is a ratio of a total number of all the first pixel points contained in the target smoke segmentation region to a total number of all pixel points in the motion region; the second area ratio ratio is a ratio of a total number of all the second pixel points contained in the target flame segmentation region to the total number of all pixel points in the motion region.

6. A device for detecting pyrotechnic charges, characterized in that including: a first acquisition module, configured to acquire a plurality of continuous images from video image data; a motion detection module, configured to perform motion detection on each image to obtain a motion region of each image; an identification module, configured to input the motion region into an image semantic segmentation neural network model to obtain a suspected smoke segmentation region and a suspected flame segmentation region of each image through identification; a second acquisition module, configured to acquire a target smoke segmentation region in the suspected smoke segmentation region and a target flame segmentation region in the suspected flame segmentation region respectively; the target smoke segmentation region is a region surrounded by all the first pixel points that fall in the suspected smoke segmentation region and the motion region at the same time; and the target flame segmentation region is a region surrounded by all the second pixel points that fall in the suspected flame segmentation region and the motion region at the same time; a calculation module, configured to calculate a first area ratio ratio and a second area ratio ratio of the target smoke segmentation region and the target flame segmentation region in the motion region respectively; a first determination module, configured to determine that the target smoke segmentation region and the target flame segmentation region are real smoke regions and real flame regions respectively when the first area ratio ratio and the second area ratio ratio are both greater than or equal to a first preset threshold; a second determination module, configured to generate a firework alarm if the real smoke region and the real flame region exist in each image and there is an overlapping part between the real smoke region and the real flame region; the target smoke segmentation region in the suspected smoke segmentation region and the target flame segmentation region in the suspected flame segmentation region are acquired respectively, including: coordinates of each pixel point contained in each region of the suspected smoke segmentation region, the suspected flame segmentation region and the motion region are acquired respectively; the pixel points with the same coordinate values in the suspected smoke segmentation region and the motion region are taken as the first pixel points, and the target smoke segmentation region is generated according to all the first pixel points; The second pixel points are pixel points with the same coordinate values in the suspected flame segmentation area and the motion area, and the target flame segmentation area is generated according to all the second pixel points.

7. A device for detecting a pyrotechnic charge according to claim 6, characterised in that The second determining module further includes: The first determining unit is configured to generate a fire and smoke alarm if the real smoke area or the real flame area exists in the continuous multiple frames of images.

8. A device for detecting a pyrotechnic charge according to claim 6, wherein The identification module includes: The first identification unit is configured to identify each pixel point of the motion area of each frame of image by using an image semantic segmentation neural network model to obtain probability values of each pixel point being light smoke, thick smoke, small fire or large fire respectively. The second determining unit is configured to determine the type of each pixel point according to the maximum probability value of each pixel point. The first integration unit is configured to integrate the pixel points of the light smoke type and the thick smoke type to obtain the suspected smoke segmentation area. The second integration unit is configured to integrate the pixel points of the small fire type and the large fire type to obtain the suspected flame segmentation area.

9. A device for detecting a pyrotechnic charge according to claim 8, characterised in that, The identification module further includes: The third integration unit is configured to combine the suspected smoke segmentation areas of each of the frame of image and the two adjacent frames of image of the frame of image to obtain the suspected smoke segmentation area of the frame of image. The fourth integration unit is configured to combine the suspected flame segmentation areas of each of the frame of image and the two adjacent frames of image of the frame of image to obtain the suspected flame segmentation area of the frame of image.

Images