A method, device, equipment and medium for detecting fireworks

Abstract

The present application discloses a fireworks detection method, device, equipment and medium, including: obtaining a current frame image, and extracting a target frame image from a video to be detected that satisfies a preset number of interval frames with the current frame image; obtaining the optical flow information of the pixel points corresponding to the current frame image and the target frame image in the coordinate axis direction of a two-dimensional coordinate system, and generating a corresponding optical flow information map; constructing a high-dimensional map based on the optical flow information map and the current frame image, and inputting the current frame image, the optical flow information map, and the high-dimensional map into a detection network to determine a target detection area similar to the fireworks feature information; using a predefined static target filtering method to detect the target detection area, and determining whether the target object in the target detection area is in a static state. If the target object is in a static state, the target detection area corresponding to the target object is removed. By constructing a high-dimensional map and filtering static targets, the detection rate is improved and false detection is reduced, which is especially applicable to infrared scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and recognition, and particularly to a method, device, equipment and medium for detecting fireworks. Background Art

[0002] With the continuous popularization of industrial automation, the concept of safe production has been continuously mentioned. Especially for fire prevention in gas stations, cotton mills, etc., any small spark may cause a major accident. Therefore, the control of open flames is particularly important.

[0003] Currently, on the one hand, early warnings are carried out by using smoke alarms, but the equipment is prone to problems such as aging or damage due to long-term disuse. On the other hand, fireworks are detected based on video images. However, most of the existing technologies on the market use color image scenes and rely on color information, which are not applicable to grayscale images in the infrared scene. For the grayscale information in infrared images, generally, the grayscale information is directly extracted or converted into a pseudo-color image, but both of these methods have certain misdetections.

[0004] In summary, how to detect fireworks in grayscale images in the infrared scene and reduce misdetection is a problem to be solved at present. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a method, device, equipment and medium for detecting fireworks, which can detect fireworks in grayscale images in the infrared scene and reduce misdetection. The specific solutions are as follows:

[0006] In the first aspect, the present application discloses a method for detecting fireworks, including:

[0007] Obtain the current frame image, and extract the target frame image that satisfies a preset number of frame intervals from the video to be detected and corresponds to the current frame image;

[0008] Obtain the optical flow information of the corresponding pixel points of the current frame image and the target frame image in the coordinate axis direction of the two-dimensional coordinate system, and generate a corresponding optical flow information map;

[0009] Construct a high-dimensional map based on the optical flow information map and the current frame image, and input the current frame image, the optical flow information map, and the high-dimensional map into a detection network to determine a target detection area similar to the fireworks feature information;

[0010] Detect the target detection area by using a predefined static target filtering method, and judge whether the target object in the target detection area is in a static state. If the target object is in a static state, then remove the target detection area corresponding to the target object.

[0011] Optionally, obtaining the optical flow information of the corresponding pixel points of the current frame image and the target frame image in the coordinate axis directions of the two-dimensional coordinate system, and generating a corresponding optical flow information map, includes:

[0012] Obtaining the optical flow information of the corresponding pixel points of the current frame image and the target frame image in the x direction and y direction of the two-dimensional coordinate system, and generating an x-direction optical flow information map and a y-direction optical flow information map;

[0013] Correspondingly, constructing a high-dimensional map based on the optical flow information map and the current frame image, includes:

[0014] Constructing a high-dimensional map based on the x-direction optical flow information map, the y-direction optical flow information map, the current frame image, and their corresponding weight coefficients.

[0015] Optionally, before constructing a high-dimensional map based on the x-direction optical flow information map, the y-direction optical flow information map, the current frame image, and their corresponding weight coefficients, further includes:

[0016] Determining a first weight coefficient for the x-direction optical flow information map; determining a second weight coefficient for the y-direction optical flow information map and a third weight coefficient for the current frame image; wherein, the first weight coefficient is the same as the second weight coefficient, and the sum of the first weight coefficient, the second weight coefficient, and the third weight coefficient is equal to 1.

[0017] Optionally, determining the third weight coefficient of the current frame image, includes:

[0018] Constructing a comparison image including the current frame image and a fourth weight coefficient;

[0019] Using the structural similarity method to compare the current frame image and the comparison image to obtain a comparison value;

[0020] When the comparison value exceeds a preset threshold, determining the fourth weight coefficient as the third weight coefficient of the current frame image.

[0021] Optionally, inputting the current frame image, the optical flow information map, and the high-dimensional map into a detection network to determine a target detection area similar to the fireworks feature information, includes:

[0022] Inputting the current frame image, the optical flow information map, and the high-dimensional map into the first branch network, the second branch network, and the third branch network in the YOLO detection network respectively; wherein, the corresponding channel numbers in the first branch network, the second branch network, and the third branch network satisfy a preset ratio, and the preset ratio is determined based on the composite ratio relationship of the input images of each branch network;

[0023] Fuse the first feature information extracted by the first branch network, the second feature information extracted by the second branch network, and the third feature information extracted by the third branch network to obtain the fused feature information;

[0024] Use a convolutional layer to process the fused feature information, and perform classification and regression through a detection head to determine the target detection region similar to the fireworks feature information.

[0025] Optionally, detecting the target detection region using a predefined static target filtering method and determining whether the target object in the target detection region is in a stationary state includes:

[0026] Extract the previous frame image of the current frame image from the video to be detected, and screen out the region position corresponding to the target detection region from the previous frame image;

[0027] Extract a first data block from the target detection region, and extract a second data block from the region position;

[0028] Obtain the optical flow information of the corresponding data points in the first data block and the second data block, generate a corresponding optical flow data map, and determine whether the target object in the target detection region is in a stationary state based on the optical flow data map.

[0029] Optionally, determining whether the target object in the target detection region is in a stationary state based on the optical flow data map includes:

[0030] Perform sampling processing on the optical flow data map at a preset sampling point interval to obtain a sampled optical flow data map;

[0031] Input the sampled optical flow data map into a classifier to determine whether the target object in the target detection region is in a stationary state.

[0032] In a second aspect, the present application discloses a fireworks detection device, including:

[0033] A video frame acquisition module for acquiring a current frame image and extracting a target frame image that satisfies a preset number of frame intervals from the video to be detected;

[0034] An optical flow information acquisition module for acquiring the optical flow information of the corresponding pixel points of the current frame image and the target frame image in the coordinate axis directions of a two-dimensional coordinate system, and generating a corresponding optical flow information map;

[0035] A detection module, configured to construct a high-dimensional map based on the optical flow information map and the current frame image, and input the current frame image, the optical flow information map, and the high-dimensional map into a detection network to determine a target detection area similar to the fireworks feature information;

[0036] An elimination module, configured to detect the target detection area by using a predefined static target filtering method, and determine whether the target object in the target detection area is in a static state. If the target object is in a static state, the target detection area corresponding to the target object is eliminated.

[0037] In a third aspect, the present application discloses an electronic device, including:

[0038] A memory, configured to store a computer program;

[0039] A processor, configured to execute the computer program to implement the steps of the aforementioned fireworks detection method.

[0040] In a fourth aspect, the present application discloses a computer-readable storage medium, configured to store a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned fireworks detection method are implemented.

[0041] It can be seen that the present application first obtains a current frame image, and extracts a target frame image that satisfies a preset number of frame intervals from a video to be detected; obtains optical flow information of corresponding pixel points of the current frame image and the target frame image in the coordinate axis direction of a two-dimensional coordinate system, and generates a corresponding optical flow information map; constructs a high-dimensional map based on the optical flow information map and the current frame image, and inputs the current frame image, the optical flow information map, and the high-dimensional map into a detection network to determine a target detection area similar to the fireworks feature information; detects the target detection area by using a predefined static target filtering method, and determines whether the target object in the target detection area is in a static state. If the target object is in a static state, the target detection area corresponding to the target object is eliminated. Thus, it can be seen that the present application obtains the optical flow information between the corresponding pixel points of the current frame image and the target frame image to generate an optical flow information map representing the target motion state, constructs a high-dimensional map in combination with the current frame image, then inputs the current frame image, the optical flow information map, and the high-dimensional map into a detection network for detection to obtain a target detection area, and finally determines whether the target object is in a static state by using a predefined static target filtering method to distinguish between static targets and moving targets, and eliminates the detection area where the static target is located. Through the above technical solution, it is possible to detect fireworks in a gray-scale image in an infrared scenario, and improve the detection rate and reduce false detection. Description of the Drawings

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on the provided drawings.

[0043] Figure 1 Flowchart of a fireworks detection method disclosed in this application;

[0044] Figure 2a Schematic diagram of an original image disclosed in this application;

[0045] Figure 2b Schematic diagram of a high-dimensional spectrum diagram disclosed in this application;

[0046] Figure 2c Schematic diagram of the marking of the beating difference points in the high-dimensional spectrum diagram disclosed in this application;

[0047] Figure 3a Schematic diagram of the optical flow data diagram of a stationary target disclosed in this application;

[0048] Figure 3b Schematic diagram of the optical flow data diagram of a moving target disclosed in this application

[0049] Figure 4 Flowchart of a specific fireworks detection method disclosed in this application;

[0050] Figure 5 Flowchart of a specific fireworks detection method disclosed in this application;

[0051] Figure 6 Schematic diagram of the original detection network model disclosed in this application;

[0052] Figure 7 Schematic diagram of the modified detection network model disclosed in this application;

[0053] Figure 8 Schematic diagram of a one-layer convolution model disclosed in this application;

[0054] Figure 9 Schematic diagram of the structure of a fireworks detection device disclosed in this application;

[0055] Figure 10 Schematic diagram of the structure of an electronic device disclosed in this application. Detailed implementation manners

[0056] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0057] Currently, for the gray-scale information in infrared images, generally, the gray-scale information is directly extracted or converted into a pseudo-color image, but both of these two methods have certain misdetections. For this reason, the embodiments of the present application disclose a method, device, equipment and medium for detecting fireworks, which can detect fireworks in gray-scale images in an infrared scene and reduce misdetections.

[0058] See Figure 1 As shown, the embodiments of the present application disclose a method for detecting fireworks, and the method includes:

[0059] Step S11: Obtain the current frame image, and extract a target frame image from the video to be detected that satisfies a preset number of frame intervals with the current frame image.

[0060] In this embodiment, first, the current frame image is obtained, and a target frame image that satisfies a preset number of frame intervals with the current frame image is extracted from the video to be detected. It should be noted that the value range of the above-mentioned preset number of frame intervals is from 1 to 5, that is, the number of frame intervals between the extracted target frame image and the current frame image differs by 1 to 5 frames.

[0061] Step S12: Obtain the optical flow information of the corresponding pixel points of the current frame image and the target frame image in the coordinate axis direction of the two-dimensional coordinate system, and generate a corresponding optical flow information map.

[0062] In this embodiment, the optical flow method is used to calculate the optical flow information of the corresponding pixel points of the current frame image and the target frame image on the coordinate axis in the two-dimensional coordinate system, and a corresponding optical flow information map is generated. In this embodiment, the motion state of the target is captured by calculating the optical flow information of the front and rear frames using the optical flow method.

[0063] Step S13: Construct a high-dimensional map based on the optical flow information map and the current frame image, and input the current frame image, the optical flow information map, and the high-dimensional map into a detection network to determine a target detection area similar to the fireworks feature information.

[0064] In this embodiment, after obtaining the optical flow information map, a high-dimensional map is constructed in combination with the current frame image, and the current frame image, the optical flow information map, and the high-dimensional map are input into a detection network for detection to determine a target detection area similar to the firework feature information. It can be understood that the current frame image is used to represent grayscale information, and the optical flow information map is used to represent motion state information. The embodiment of the present application realizes a method for fusing grayscale information and firework optical flow motion state information in an infrared scene to construct a multi-dimensional firework map for detection, and solves the problem of converting a grayscale image into a multi-dimensional information image in an infrared scene. Figure 2a and Figure 2b are schematic diagrams of the original image and the high-dimensional map respectively. Figure 2c is Figure 2b a schematic diagram in which beating difference points are marked on the basis of the high-dimensional map in

[0065] Step S14: Detect the target detection area by using a predefined static target filtering method, and determine whether the target object in the target detection area is in a static state. If the target object is in a static state, the target detection area corresponding to the target object is removed.

[0066] In this embodiment, detecting the target detection area by using a predefined static target filtering method and determining whether the target object in the target detection area is in a static state may include: extracting the previous frame image of the current frame image from the video to be detected, and screening out the area position corresponding to the target detection area from the previous frame image; extracting a first data block from the target detection area, and extracting a second data block from the area position; obtaining the optical flow information of the corresponding data points in the first data block and the second data block, generating a corresponding optical flow data map, and determining whether the target object in the target detection area is in a static state based on the optical flow data map. It can be understood that in this embodiment, it is necessary to extract the previous frame image of the current frame image from the video to be detected, and denote the current frame image as Gray i , and denote the previous frame image as PreImg i , then screen out the ROI area (region of interest) from PreImg i , where the ROI area refers to the area position corresponding to the target detection area in Gray i ; then extract a first data block from the target detection area, denoted as ROI now , and extract a second data block from the area position, denoted as ROI pre, and perform optical flow analysis on the first data block and the second data block to generate an optical flow data graph, and determine whether the target object in the target detection area is in a stationary state through the optical flow data graph.

[0067] The above determination of whether the target object in the target detection area is in a stationary state based on the optical flow data graph includes: sampling the optical flow data graph at a preset sampling point interval to obtain a sampled optical flow data graph; inputting the sampled optical flow data graph into a classifier to determine whether the target object in the target detection area is in a stationary state. It can be understood that in this embodiment, it is necessary to perform sampling point processing on the obtained optical flow data graph to obtain a sampled optical flow data graph. The above preset sampling point interval can be set to 10, that is, a sampling point is obtained every 10 points. Figure 3a and Figure 3b respectively disclose the optical flow data graph of the stationary target and the optical flow data graph of the moving target after sampling, and finally input these two sampled optical flow data graphs into a classifier to determine whether the target object in the target detection area is in a stationary state. Through the above technical solution, the problem of false detection introduced by stationary overexposed sources and the like can be reduced.

[0068] It can be seen that this application first obtains the current frame image, and extracts the target frame image that satisfies the preset number of frame intervals from the video to be detected; obtains the optical flow information of the corresponding pixel points of the current frame image and the target frame image in the coordinate axis direction of the two-dimensional coordinate system, and generates a corresponding optical flow information graph; constructs a high-dimensional graph based on the optical flow information graph and the current frame image, and inputs the current frame image, the optical flow information graph, and the high-dimensional graph into the detection network to determine the target detection area similar to the fireworks feature information; uses a predefined static target filtering method to detect the target detection area, and determines whether the target object in the target detection area is in a stationary state. If the target object is in a stationary state, the target detection area corresponding to the target object is excluded. Thus, this application generates an optical flow information graph representing the target motion state by obtaining the optical flow information between the corresponding pixel points of the current frame image and the target frame image, constructs a high-dimensional graph in combination with the current frame image, then inputs the current frame image, the optical flow information graph, and the high-dimensional graph into the detection network for detection to obtain the target detection area, and finally determines whether the target object is in a stationary state through a predefined static target filtering method to distinguish between stationary targets and moving targets, and excludes the detection area where the stationary target is located. Through the above technical solution, it is possible to detect fireworks in a gray-scale image in an infrared scene, improve the detection rate, and reduce false detection.

[0069] See Figure 4As shown, an embodiment of the present application discloses a specific fireworks detection method. Compared with the previous embodiment, the technical solution in this embodiment is further described and optimized. Specifically, it includes:

[0070] Step S21: Obtain the current frame image, and extract a target frame image from the video to be detected that satisfies a preset number of frame intervals with the current frame image.

[0071] Step S22: Obtain the optical flow information of the corresponding pixel points of the current frame image and the target frame image in the x - direction and y - direction in the two - dimensional coordinate system, and generate an x - direction optical flow information map and a y - direction optical flow information map.

[0072] In this embodiment, the optical flow method is used to calculate the optical flow information of the corresponding pixel points of the current frame image and the target frame image in the x - direction and y - direction in the two - dimensional coordinate system, and generate an x - direction optical flow information map and a y - direction optical flow information map.

[0073] Step S23: Determine the first weight coefficient of the x - direction optical flow information map; determine the second weight coefficient of the y - direction optical flow information map and the third weight coefficient of the current frame image; wherein, the first weight coefficient is the same as the second weight coefficient, and the sum of the first weight coefficient, the second weight coefficient, and the third weight coefficient is equal to 1.

[0074] In this embodiment, it is necessary to determine the first weight coefficient α of the x - direction optical flow information map, the second weight coefficient β of the y - direction optical flow information map, and the third weight coefficient γ of the current frame image to construct a high - dimensional spectrum. It should be noted that α and β are the weights of optical flow maps in different directions, so they have the same value, that is, the first weight coefficient α and the second weight coefficient β are the same, and it is also necessary to satisfy that the sum of α, β, and γ is 1, that is:

[0075] α + β + γ = 1;

[0076] Determining the third weight coefficient of the current frame image includes: constructing a comparison image including the current frame image and a fourth weight coefficient; comparing the current frame image and the comparison image using the structural similarity method to obtain a comparison value; when the comparison value exceeds a preset threshold, determining the fourth weight coefficient as the third weight coefficient of the current frame image. It should be noted that γ is the weight of the grayscale image. When obtaining the γ value, it is necessary to ensure that the information of the grayscale image is not lost too much, otherwise key information will be lost. Therefore, in this embodiment, a comparison image including the current frame image and the fourth weight coefficient is constructed, and the current frame image and the comparison image are compared using the Structural Similarity (SSIM) method, and it is determined whether the comparison value exceeds the preset threshold. When the comparison value is less than the preset threshold, the comparison image at this time is considered untrustworthy. In this embodiment, the above preset threshold is set to 0.6. It should be noted that SSIM is a full-reference image quality evaluation index, which measures image similarity from three aspects: brightness, contrast, and structure. The formula is as follows:

[0077]

[0078] where i represents the current frame image and j represents the comparison image; u i and u j represent the means of the current frame image and the comparison image respectively; and represent the variances of the current frame image and the comparison image respectively, and σ ij represents the covariance; c1 and c2 are two constants, where c1 = (0.01 * 255) 2 and c2 = (0.03 * 255) 2 .

[0079] The value of the above SSIM is between 0 and 1, and the larger the value, the more similar. In this embodiment, α = 0.3, β = 0.3, and γ = 0.4 are used.

[0080] Step S24: Construct a high-dimensional map based on the x-direction optical flow information map, the y-direction optical flow information map, the current frame image, and its corresponding weight coefficients, and input the current frame image, the x-direction optical flow information map, the y-direction optical flow information map, and the high-dimensional map into the detection network to determine a target detection area similar to the fireworks feature information.

[0081] In this embodiment, the construction formula of the above high-dimensional map is:

[0082] Himg i = αFlow_x i + βFlow_y i + γGrayi ;

[0083] Among them, Himg i is a high-dimensional spectrum, Flow_x i is the optical flow information map in the x direction, Flow_y i is the optical flow information map in the y direction, and Gray i is the current frame image.

[0084] Step S25: Detect the target detection area by using a predefined static target filtering method, and determine whether the target object in the target detection area is in a stationary state. If the target object is in a stationary state, then eliminate the target detection area corresponding to the target object.

[0085] Among them, for the more specific processing procedures of the above steps S21, S24, and S25, reference can be made to the corresponding content disclosed in the foregoing embodiments, and details will not be elaborated here.

[0086] It can be seen that before constructing the high-dimensional spectrum in the embodiment of the present application, it is necessary to determine the specific weight coefficient values corresponding to the current frame image, the optical flow information map in the x direction, and the optical flow information map in the y direction. And when determining the third weight coefficient of the current frame image, it is necessary to use the structural similarity method for evaluation to obtain the third weight coefficient that meets the preset threshold conditions, so as to avoid losing too much key information in the grayscale image.

[0087] See Figure 5 As shown, the embodiment of the present application discloses a specific fireworks detection method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically, it includes:

[0088] Step S31: Obtain the current frame image, and extract the target frame image that satisfies the preset number of frames interval from the video to be detected.

[0089] Step S32: Obtain the optical flow information of the pixel points corresponding to the current frame image and the target frame image in the coordinate axis directions of the two-dimensional coordinate system, and generate the corresponding optical flow information map.

[0090] Step S33: Construct a high-dimensional spectrum based on the optical flow information map and the current frame image, and input the current frame image, the optical flow information map, and the high-dimensional spectrum into the first branch network, the second branch network, and the third branch network in the YOLO detection network respectively; among them, the corresponding number of channels in the first branch network, the second branch network, and the third branch network satisfies a preset ratio, and the preset ratio is determined based on the synthesis ratio relationship of the input images of each branch network.

[0091] In this embodiment, the detection network used is YOLO (i.e., You Only Look Once), and the original backbone network is as follows Figure 6 shown. In this embodiment, the backbone network is modified accordingly. By dividing the original single-image input into three branch networks for input, the structures of each branch network are similar. The modified detection network is as follows Figure 7 shown. Specifically, the current frame image is input into the first branch network, the optical flow information map is input into the second branch network, and the high-dimensional spectral map is input into the third branch network. Among them, the first branch network is used to extract the texture information of the current frame image, that is, the original image, as the first feature information; the second branch network is used to extract the motion information in the optical flow information map as the second feature information; the third branch network is used to extract the high-dimensional unknown useful information as the third feature information. It should also be noted that the corresponding number of channels in the first branch network, the second branch network, and the third branch network satisfies a preset ratio. Based on the complexity of the data features, the number of channels of the three branch networks is different, and the number of channels from less to more is the current frame image, the optical flow information map, and the high-dimensional spectral map in turn. The redesign of the backbone network can effectively extract the useful information required by each, avoiding the problem that a single branch extracts network information for multiple purposes and causes the network parameters to be underfitted.

[0092] When determining the preset ratio between the number of channels, the preset ratio can be determined based on the composite ratio relationship of the input images of each branch network. It can be understood that there is only one grayscale image for the current frame image, the optical flow information map includes the optical flow information in the x direction and the y direction, and the high-dimensional spectral map is a 3-channel high-dimensional map synthesized from the current frame image, the optical flow information map in the x direction, and the optical flow information map in the y direction. Therefore, the preset ratio relationship between the number of channels in this embodiment is set to 1:2:3.

[0093] Step S34: Fuse the first feature information extracted by the first branch network, the second feature information extracted by the second branch network, and the third feature information extracted by the third branch network to obtain the fused feature information.

[0094] In this embodiment, the first feature information extracted by the first branch network, the second feature information extracted by the second branch network, and the third feature information extracted by the third branch network are fused at the last layer to obtain the fused feature information, mainly by adding the channels.

[0095] Step S35: Process the fused feature information using a convolutional layer, and perform classification and regression through a detection head to determine the target detection area similar to the fireworks feature information.

[0096] In this embodiment, a layer of convolution is used to reduce the dimension and extract features from the fused feature information, and then regression classification is performed through a detection head (i.e., head) to determine a target detection region similar to the fireworks feature information. It should be noted that the layer of convolution consists of a 1×1 convolution layer, a batchnorm layer, an activation layer, and a 3×3 convolution layer. For details, see Figure 8 as shown.

[0097] Step S36: Use a predefined static object filtering method to detect the target detection region, and determine whether the target object in the target detection region is in a stationary state. If the target object is in a stationary state, then eliminate the target detection region corresponding to the target object.

[0098] Among them, the more specific processing procedures for the above steps S31, S32, and S36 can refer to the corresponding content disclosed in the foregoing embodiments, and will not be elaborated here.

[0099] It can be seen that in the embodiment of the present application, by modifying the backbone network of the detection network, the original single-image input is divided into three branch networks for input. Specifically, the current frame image is input into the first branch network, the optical flow information map is input into the second branch network, and the high-dimensional spectral map is input into the third branch network. Moreover, the number of channels in each branch network satisfies a preset proportional relationship. Thus, the embodiment of the present application realizes the method of hierarchical processing of different information flows by the backbone network and the method of selecting the number of channels between layers.

[0100] See Figure 9 As shown, the embodiment of the present application discloses a fireworks detection device, including:

[0101] A video frame acquisition module 11, configured to acquire a current frame image and extract a target frame image that satisfies a preset number of frames at intervals from the video to be detected;

[0102] An optical flow information acquisition module 12, configured to acquire the optical flow information of the pixel points corresponding to the current frame image and the target frame image in the coordinate axis directions of the two-dimensional coordinate system, and generate a corresponding optical flow information map;

[0103] A detection module 13, configured to construct a high-dimensional spectral map based on the optical flow information map and the current frame image, and input the current frame image, the optical flow information map, and the high-dimensional spectral map into a detection network to determine a target detection region similar to the fireworks feature information;

[0104] The elimination module 14 is used to detect the target detection area by using a pre-defined static target filtering method, and determine whether the target object in the target detection area is in a static state. If the target object is in a static state, the target detection area corresponding to the target object is eliminated.

[0105] It can be seen that this application first obtains the current frame image, and extracts the target frame image that satisfies the preset number of frame intervals from the video to be detected; obtains the optical flow information of the corresponding pixel points of the current frame image and the target frame image in the coordinate axis direction of the two-dimensional coordinate system, and generates a corresponding optical flow information map; constructs a high-dimensional map based on the optical flow information map and the current frame image, and inputs the current frame image, the optical flow information map, and the high-dimensional map into the detection network to determine the target detection area similar to the fireworks feature information; uses a pre-defined static target filtering method to detect the target detection area, and determines whether the target object in the target detection area is in a static state. If the target object is in a static state, the target detection area corresponding to the target object is eliminated. Thus, this application obtains the optical flow information between the corresponding pixel points of the current frame image and the target frame image to generate an optical flow information map representing the target motion state, constructs a high-dimensional map in combination with the current frame image, then inputs the current frame image, the optical flow information map, and the high-dimensional map into the detection network for detection to obtain the target detection area, and finally uses a pre-defined static target filtering method to determine whether the target object is in a static state to distinguish between static targets and moving targets, and eliminate the detection area where the static target is located. Through the above technical solutions, it is possible to detect fireworks in grayscale images in an infrared scenario, and improve the detection rate and reduce false detection.

[0106] Figure 10 It is a schematic structural diagram of an electronic device provided by an embodiment of this application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the fireworks detection method executed by the computer device disclosed in any of the foregoing embodiments.

[0107] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the computer device 20; the communication interface 24 can create a data transmission channel between the computer device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and no specific limitation is imposed here; the input / output interface 25 is used to obtain external input data or output data to the outside, and the specific interface type can be selected according to specific application needs, and no specific limitation is imposed here.

[0108] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor used to process data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for the rendering and drawing of the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0109] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, a random access memory, a disk, or an optical disc, etc. The resources stored thereon include an operating system 221, a computer program 222, data 223, etc., and the storage method can be temporary storage or permanent storage.

[0110] Among them, the operating system 221 is used to manage and control each hardware device on the computer device 20 and the computer program 222, so as to implement the operation and processing of the massive data 223 in the memory 22 by the processor 21. It can be Windows, Unix, Linux, etc. In addition to the computer program that can be used to complete the fire detection method executed by the computer device 20 disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks. The data 223 may include not only the data transmitted by the external device received by the computer device, but also the data collected by its own input / output interface 25, etc.

[0111] Further, an embodiment of the present application also discloses a storage medium in which a computer program is stored. When the computer program is loaded and executed by a processor, the method steps executed during the fire detection process disclosed in any of the foregoing embodiments are implemented.

[0112] In this specification, the various embodiments are described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and reference can be made to the description of the method part for related parts.

[0113] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0114] The steps of the method or algorithm described in combination with the embodiments disclosed in this article can be directly implemented by hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0115] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent in such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0116] The above has introduced in detail a fireworks detection method, device, equipment and storage medium provided by the present invention. Specific examples are used in this text to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A method for detecting fireworks, characterized in that, Including: Obtain the current frame image, and extract a target frame image that satisfies a preset number of frame intervals from the video to be detected with respect to the current frame image; Obtain the optical flow information of the corresponding pixel points of the current frame image and the target frame image in the axis directions of a two-dimensional coordinate system, and generate a corresponding optical flow information map; Construct a high-dimensional map based on the optical flow information map and the current frame image, and input the current frame image, the optical flow information map, and the high-dimensional map into a detection network to determine a target detection area similar to the fireworks feature information; Detect the target detection area using a predefined static target filtering method, and determine whether the target object in the target detection area is in a stationary state. If the target object is in a stationary state, then eliminate the target detection area corresponding to the target object; Wherein, the obtaining the optical flow information of the corresponding pixel points of the current frame image and the target frame image in the axis directions of a two-dimensional coordinate system, and generating a corresponding optical flow information map, includes: Obtain the optical flow information of the corresponding pixel points of the current frame image and the target frame image in the x direction and the y direction of a two-dimensional coordinate system, and generate an x-direction optical flow information map and a y-direction optical flow information map; Correspondingly, the constructing a high-dimensional map based on the optical flow information map and the current frame image includes: Construct a high-dimensional map based on the x-direction optical flow information map, the y-direction optical flow information map, the current frame image, and their corresponding weight coefficients.

2. The method for detecting fireworks according to claim 1, characterized in that, Before the constructing a high-dimensional map based on the x-direction optical flow information map, the y-direction optical flow information map, the current frame image, and their corresponding weight coefficients, it further includes: Determine a first weight coefficient of the x-direction optical flow information map; determine a second weight coefficient of the y-direction optical flow information map and a third weight coefficient of the current frame image; wherein, the first weight coefficient is the same as the second weight coefficient, and the sum of the first weight coefficient, the second weight coefficient, and the third weight coefficient is equal to 1.

3. The method for detecting fireworks according to claim 2, characterized in that, The determining the third weight coefficient of the current frame image includes: Construct a comparison image including the current frame image and a fourth weight coefficient; Use the structural similarity method to compare the current frame image and the comparison image to obtain a comparison value; When the comparison value exceeds a preset threshold, then determine the fourth weight coefficient as the third weight coefficient of the current frame image.

4. The method for detecting fireworks according to claim 1, characterized in that, The inputting the current frame image, the optical flow information map, and the high-dimensional map into a detection network to determine a target detection area similar to the fireworks feature information includes: Input the current frame image, the optical flow information map, and the high-dimensional map into the first branch network, the second branch network, and the third branch network in the YOLO detection network respectively; wherein, the corresponding number of channels in the first branch network, the second branch network, and the third branch network satisfies a preset ratio, and the preset ratio is determined based on the composite ratio relationship of the input images of each branch network; Fuse the first feature information extracted by the first branch network, the second feature information extracted by the second branch network, and the third feature information extracted by the third branch network to obtain the fused feature information; Process the fused feature information using a convolutional layer, and perform classification and regression through a detection head to determine a target detection region similar to the fireworks feature information.

5. The method for detecting fireworks according to any one of claims 1 to 4, characterized in that, The detecting the target detection region using a predefined static target filtering method and determining whether the target object in the target detection region is in a stationary state includes: Extract the previous frame image of the current frame image from the video to be detected, and screen out the region position corresponding to the target detection region from the previous frame image; Extract a first data block from the target detection region, and extract a second data block from the region position; Obtain the optical flow information of the corresponding data points in the first data block and the second data block, generate a corresponding optical flow data map, and determine whether the target object in the target detection region is in a stationary state based on the optical flow data map.

6. The method for detecting fireworks according to claim 5, characterized in that, The determining whether the target object in the target detection region is in a stationary state based on the optical flow data map includes: Perform sampling processing on the optical flow data map at a preset sampling point interval to obtain a sampled optical flow data map; Input the sampled optical flow data map into a classifier to determine whether the target object in the target detection region is in a stationary state.

7. A fireworks detection device, characterized in that, including: A video frame acquisition module, configured to acquire a current frame image, and extract a target frame image that satisfies a preset number of frame intervals from the video to be detected; An optical flow information acquisition module, configured to acquire the optical flow information of the corresponding pixel points of the current frame image and the target frame image in the coordinate axis directions of a two-dimensional coordinate system, and generate a corresponding optical flow information map; A detection module, configured to construct a high-dimensional map based on the optical flow information map and the current frame image, and input the current frame image, the optical flow information map, and the high-dimensional map into a detection network to determine a target detection region similar to the fireworks feature information; An elimination module, configured to detect the target detection region using a predefined static target filtering method, and determine whether the target object in the target detection region is in a stationary state. If the target object is in a stationary state, eliminate the target detection region corresponding to the target object; Among them, the optical flow information acquisition module is specifically configured to acquire the optical flow information of the corresponding pixel points of the current frame image and the target frame image in the x direction and the y direction of a two-dimensional coordinate system, and generate an x-direction optical flow information map and a y-direction optical flow information map; Correspondingly, the detection module is specifically configured to construct a high-dimensional map based on the x-direction optical flow information map, the y-direction optical flow information map, the current frame image, and their corresponding weight coefficients.

8. An electronic device, characterized in that, including: A memory, configured to store a computer program; A processor, configured to execute the computer program to implement the steps of the fireworks detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, For storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the fireworks detection method according to any one of claims 1 to 6.

Images