Smoke and fire identification method and device, electronic equipment and storage medium

Abstract

The invention provides a smoke and fire identification method and device, electronic equipment and a storage medium, and belongs to the field of smoke and fire identification. The problem that a traditional smoke and fire recognition system is low in accuracy is solved. The smoke and fire identification method comprises the following steps: acquiring smoke and fire image data and marking the image data to construct a smoke and fire data set; dividing the firework data set into a training set for establishing a model, a verification set for parameter adjustment and optimization of the model and a test set for evaluating the performance of the model; constructing a smoke and fire identification model on the basis of a convolutional neural network; using the training set to train the smoke and fire identification model, and adjusting hyper-parameters of the smoke and fire identification model through the verification set; and acquiring a video stream by using camera equipment, reasoning and judging the video stream by applying the trained smoke and fire identification model, and detecting whether an ignition point exists in the monitoring area. The device is mainly used for identifying smoke and fire alarm.

Description

Technical Field

[0001] The present invention belongs to the field of fireworks recognition, and in particular relates to a fireworks recognition method, device, electronic equipment and storage medium. Background Art

[0002] With the development of modern society, public safety issues have become increasingly prominent, especially in large gatherings, celebrations, transportation hubs and other places, the detection and identification of dangerous items such as fireworks is particularly important.

[0003] The existing fireworks recognition system has some defects. First, there is a significant delay when processing large amounts of real-time data, which cannot meet the high real-time requirements of the modern public safety field. Second, the performance is unstable under different environmental conditions. For example, the recognition accuracy and stability will drop significantly in extreme weather. Third, when facing fireworks detection tasks in complex environments, the false detection rate and missed detection rate are high, and the terminal deployment uses Jetson NX or Jetson Nano devices produced abroad, which is expensive and not conducive to large-scale deployment. The core technology is controlled by others. Summary of the invention

[0004] In view of this, the present invention aims to provide a fireworks recognition method, device, electronic device and storage medium to solve the problem of low accuracy of traditional fireworks recognition systems.

[0005] To achieve the above object, the present invention adopts the following technical solution. According to a first aspect of the present invention, a method for identifying fireworks is provided, comprising the following steps:

[0006] Obtain fireworks image data and annotate the image data to build a fireworks dataset;

[0007] The fireworks dataset is divided into a training set for model building, a validation set for model parameter adjustment and optimization, and a test set for evaluating model performance.

[0008] Use convolutional neural network as the basis to build a fireworks recognition model;

[0009] Use the training set to train the fireworks recognition model, and use the validation set to adjust the hyperparameters of the fireworks recognition model;

[0010] Use the camera to obtain the video stream, apply the trained fireworks recognition model to infer and judge the video stream, and detect whether there is a fire point in the monitoring area.

[0011] Furthermore, mosaic data augmentation is used to expand the fireworks dataset when constructing the fireworks dataset.

[0012] Furthermore, the ratio of the training set, validation set and test set is 8:1:1.

[0013] Furthermore, the fireworks recognition model is constructed as follows: the baseline model uses the YOLOv8 algorithm, introduces polarized self-attention (PSA) to weight image features, designs a cross-stage residual module based on the idea of ​​CSPNet (Cross Stage Partial Networks) and feature reuse, and uses Ghost convolution processing.

[0014] Furthermore, the polarized self-attention includes a channel attention branch and a spatial attention branch, and the Softmax function and the Sigmoid function are mixed in the attention calculation of the channel attention branch and the spatial attention branch.

[0015] Furthermore, the Ghost convolution processing flow is: first, a 1*1 standard convolution is used to obtain a multi-channel feature layer, and the Ghost convolution performs a linear convolution operation on the obtained multi-channel feature layer channel by channel, and then splices and stacks them.

[0016] Furthermore, the reasoning and judgment includes the following steps: the fireworks recognition model receives the predicted image and preprocesses the image, performs preliminary feature extraction through the CSPDarknet backbone feature extraction network, integrates the preliminary extracted features through the SPPF spatial pyramid pooling structure to improve the model receptive field, the path aggregation network fuses the semantic features with the fine-grained features, the cross-stage residual module optimizes the gradient information, the polarized self-attention structure enhances the significance of the fireworks target information and suppresses background interference, the detection head outputs the predicted information, and the NMS non-maximum suppression outputs the fireworks target recognition result.

[0017] According to a second aspect of the present invention, there is provided a fireworks identification device, comprising:

[0018] A data preprocessing module is used to obtain fireworks image data and annotate the image data to construct a fireworks data set;

[0019] A data set partitioning module is used to partition the fireworks data set into a training set for building a model, a validation set for model parameter adjustment and optimization, and a test set for evaluating model performance;

[0020] Neural network model building module, used to build a fireworks recognition model based on convolutional neural network;

[0021] Model training and optimization module, used to train the fireworks recognition model using the training set and adjust the hyperparameters of the fireworks recognition model using the validation set;

[0022] The model verification module is used to use the camera equipment to obtain the video stream, apply the trained fireworks recognition model to infer and judge the video stream, and detect whether there is a fire point in the monitored area.

[0023] According to a third aspect of the present invention, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the fireworks recognition method as described in the above embodiment.

[0024] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the program is executed by a processor, the method for identifying fireworks as described above is implemented.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1. This method extracts features of smoke and flame targets through a feature extraction network structure, aggregates and enhances the features of fireworks by combining a spatial pyramid pooling structure and a path aggregation network structure, and integrates a polarized self-attention mechanism to suppress background negative samples and enhance positive samples, thereby improving the algorithm's ability to resist interference in extreme weather and complex environments;

[0027] 2. This method designs a cross-stage residual module to improve the network feature expression ability, thereby improving the ability to extract small flame features, replacing the standard convolution with Ghost convolution to reduce redundant convolution kernels, reduce the number of network parameters, and further provide support for edge deployment of the model;

[0028] 3. This method improves recognition accuracy and can accurately identify fire points in the embryonic stage at the early stage of a fire, and still has stable recognition performance in extreme weather and complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0030] Figure 1 A flowchart of a method for identifying fireworks according to the present invention;

[0031] Figure 2 A schematic block diagram of a fireworks identification device according to the present invention;

[0032] Figure 3 is a schematic diagram of the electronic device according to the present invention;

[0033] Figure 4 This is a flow chart of the fireworks recognition model training of the present invention;

[0034] Figure 5 This is a structural block diagram of the fireworks recognition model of the present invention;

[0035] Figure 6 The polarized self-attention structure diagram of the present invention;

[0036] Figure 7 This is a structural diagram of the cross-stage residual module of the present invention;

[0037] Figure 8 This is a flow chart of the Ghost convolution processing described in the present invention;

[0038] Fig. 9 This is a flow chart of the reasoning and judgment of the fireworks recognition model of the present invention;

[0039] Fig.10 The detection result of the fireworks recognition model of the present invention;

[0040] Fig.11 The edge deployment process of the fireworks recognition algorithm described in the present invention;

[0041] Fig.12 This is a block diagram of the edge chip system of the present invention;

[0042] Fig.13 This is a physical picture of the edge chip of the present invention;

[0043] Fig.14 A schematic diagram of a fireworks recognition system of an edge chip of the present invention; DETAILED DESCRIPTION

[0044] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0045] The following describes the fireworks recognition method, device, electronic device and storage medium of the embodiments of the present invention with reference to the accompanying drawings. In view of the problem of low accuracy of the traditional fireworks recognition system mentioned in the background technology center, the present invention provides a fireworks recognition method, in which the smoke and flame targets are feature extracted by a feature extraction network structure, and the fireworks features are aggregated and enhanced by combining the spatial pyramid pooling structure and the path aggregation network structure. The polarized self-attention mechanism is integrated to suppress the background negative samples and enhance the positive samples, thereby improving the algorithm's anti-interference ability in extreme weather and complex environments; a cross-stage residual module is designed to improve the network feature expression ability, thereby improving the ability to extract small flame features, replacing the standard convolution with the Ghost convolution to reduce redundant convolution kernels, reduce the network parameters, and further provide support for the edge deployment of the model. In this way, the problem of low accuracy of the traditional fireworks recognition system is solved.

[0046] Specifically, Figure 1 A flowchart of a method for identifying fireworks provided by an embodiment of the present invention is shown.

[0047] like Figure 1 As shown, the fireworks recognition method includes the following steps:

[0048] In step S101, fireworks image data is obtained and the image data is annotated to construct a fireworks dataset. The specific processing method is to collect fire videos and extract frames from the videos to obtain fireworks image data. The flames and smoke targets in the images are annotated using professional annotation tools to construct a fireworks dataset. In addition, in order to further improve the generalization of the model, mosaic data enhancement is used to expand the dataset, increase the number of small flame targets in the dataset, and provide support for early identification of flames.

[0049] In step S102, the fireworks data set is divided into a training set for building a model, a validation set for model parameter adjustment and optimization, and a test set for evaluating the final performance of the model. The ratio of the data set to the test set and the validation set is 8:1:1.

[0050] In step S103, a fireworks recognition model is constructed based on a convolutional neural network. Since the local connection and weight sharing, invariance of translation and spatial transformation of the convolutional neural network are very suitable for processing image data, the convolutional neural network is used as the basis for constructing the fireworks recognition model.

[0051] In step S104, the fireworks recognition model is trained using the training set, and the hyperparameters of the fireworks recognition model are adjusted using the validation set to avoid overfitting and underfitting. The loss value of the model is calculated by forward propagation and combined with optimization algorithms, such as stochastic gradient descent (SGD, Adam), the weight parameters of the model are continuously iterated to minimize the loss value. The loss function of the fireworks recognition algorithm is divided into two parts, Box Loss and Cls Loss. The formula of Box Loss is as follows:

[0052]

[0053] Where IoU is the intersection over union ratio, b and b gt represents the center point of the two rectangular boxes, ρ represents the Euclidean distance between the two rectangular boxes, c represents the diagonal distance of the closed area of ​​the two rectangular boxes, v is used to measure the consistency of the relative proportions of the two rectangular boxes, and α is the weight coefficient. The formula of Cls Loss is as follows:

[0054]

[0055] Where q is the intersection over union of the predicted box and the true box, IoU is the intersection of the predicted box and the true box divided by the union of the two boxes, and p is the score.

[0056] In step S105, a video stream is acquired using a camera device, and the trained fire and smoke recognition model is used to infer and judge the video stream to detect whether there is a fire point in the monitoring area. In practical applications, cameras are used for all-weather monitoring, and the trained neural network model is used to infer and judge the video stream to detect whether there is a fire point in the monitoring area.

[0057] The structure of the fireworks recognition model is as follows Figure 4 As shown in the figure. The benchmark model uses the YOLOv8 algorithm, and improves the algorithm in a targeted manner to address the difficulty in detecting flame and smoke targets and the lack of obvious characteristics in the early stages of a fire. The model is mainly composed of seven modules: GBS, Bottleneck, c2f, SPPF, CSP-GBS, PSA, and Detect.

[0058] GBS is the Ghost convolution feature extraction module, which consists of a Ghost convolution layer, a normalization layer, and a SiLU activation function. It is the basic component of the feature extraction operation and is mainly located in the backbone feature extraction part of the model. The calculation formula of Ghost convolution is as follows:

[0059] Conv2d represents a two-dimensional convolutional layer, and shortcut(input) is a shortcut connection. The meanings of the various parameters are as follows: kernel_size represents the size of the convolution kernel, stride represents the step size, groups represents the number of groups in the grouped convolution, which is used to control how the grouped convolution handles channels, expansion represents the integer expansion factor, number represents the number of convolution blocks, and shortcut represents the output through the first convolutional layer.

[0060] The Bottleneck module consists of a GBS module and an add module, which adds and fuses the input with the information extracted by two GBS modules.

[0061] c2f is a feature enhancement and fusion module, which consists of a GBS module, a Bottleneck module, a split layer, and a Concat channel splicing layer. The split layer slices the input feature map, and the Concat channel splicing layer splices the two input feature layers in the channel dimension.

[0062] SPPF is a fast spatial pyramid pooling layer, which consists of a GBS module, a Max pool layer, and a Concat channel concatenation layer. The SPPF layer receives the feature map from the previous layer as input, uses three maximum pooling layers for pooling operations, and concatenates the outputs of different pooling layers to form a fixed-length feature vector as the input of subsequent other network layers.

[0063] CSP-GBS is a cross-stage residual module, which consists of a GBL module, an add layer, and a Concat channel concatenation layer. The GBL module extracts features, the add module adds and fuses the extracted features, and the Concat layer concatenates the two feature layers in the channel dimension.

[0064] The PSA polarized self-attention module uses an orthogonal approach to obtain high resolution of channels and spaces without affecting the model speed, reducing the loss of feature information. It can simultaneously calculate attention at the channel and spatial levels to suppress the background and enhance the saliency of fireworks targets.

[0065] Detect is a detection module, which consists of a CBS module and a Ghost convolution feature extraction module. The CBS module consists of a common convolution layer, a normalization layer, and a SiLU activation function, and is used for feature extraction. The Detect module receives the extracted features and integrates them for target recognition.

[0066] The fireworks recognition model introduces polarized self-attention (PSA) to weight image features, enhance the characteristics of small flames and smoke targets, and suppress background noise characteristics, thereby improving the saliency and recognition of fireworks targets. Based on CSPNet (Cross Stage Partial Networks) and feature reuse, a cross-stage residual module is designed to improve the network's feature expression ability, thereby improving the ability to capture targets with unclear features in the early stages of a fire. Furthermore, to support the edge device deployment of the algorithm, Ghost convolution operations are used to replace traditional convolution operations, reduce the number of algorithm parameters, and reduce memory usage.

[0067] In the task of fireworks target detection, the positive samples of fireworks targets are the focus of attention, while the complex background will make it difficult for the algorithm to judge. The polarized self-attention mechanism is inspired by polarization filtering. It uses an orthogonal method to obtain high resolution of channels and spaces without affecting the speed of the model, reducing the loss of feature information. It can calculate attention at the channel and spatial levels at the same time to suppress the background and enhance the significance of fireworks targets. The polarized self-attention structure is as follows: Figure 6In addition, the Softmax function and the Sigmoid function are mixed in the attention calculation of the channel branch and the spatial branch to increase the nonlinearity, making the fitted output more delicate and close to the real output.

[0068] Polarized self-attention includes two branches: channel attention and spatial attention. In the channel attention branch, 1×1 convolution is first used to adjust the number of channels of the feature, and two paths are divided. The number of feature channels of one path is compressed to 1, and the number of feature channels of the other path is adjusted to half of the number of input feature channels. Then, the Softmax function is used to enhance the features of the path with completely compressed channels, and the features on the two paths are multiplied. Finally, channel attention is obtained through operations such as convolution and Sigmoid function. In the spatial attention branch, 1×1 convolution is first used to process the input features, and two paths are also divided. Global pooling is performed on one of the paths, the spatial dimension is compressed to 1×1, and the Softmax function is used for feature enhancement. Then, the features of the two paths are multiplied, and finally, spatial attention is obtained through operations such as convolution and Sigmoid function. Channel branch attention A ch The calculation of (X) is as follows:

[0069] A ch (X) = F SG [W z (σ1(W v (X))×F SM (σ2(W q (X))))]

[0070] Where W q , W v and W z are the weights of the 1×1 convolutional layer, σ1 and σ2 are two tensor integer operators, “X” is the matrix dot product operation, and F SM () is the Softmax function operation, F SG () is the Sigmoid function operation.

[0071] A sp (X) = F SG [σ3(F SM (σ1(F GP (W q (X))))×σ2(W v (X)))]

[0072] Where W q , W v The weights σ1, σ2, and σ3 of the 1×1 convolutional layer are three tensor integer operators, “X” is the matrix dot product operation, and F SM() is the Softmax function operation, F GP () is the global pooling operation, F SG () is the Sigmoid function operation.

[0073] Finally, the obtained channel attention and spatial attention are multiplied with the input features by channel multiplication and spatial multiplication respectively, and the output features of the two branches are fused in parallel to output the polarized self-attention PSA. p The calculation of (X) is as follows:

[0074]

[0075] Based on the ideas of CSPNet and ResNet, this paper designs a cross-stage residual module. The module structure is as follows: Figure 7 As shown. This module consists of a GBL module, an add layer, and a Concat channel concatenation layer. The GBL module consists of a Ghost convolution feature extraction module, a normalization layer, and a LeakyReLU activation function. The expression of the LeakyReLU function is as follows:

[0076] f(x)=max(αx,x)

[0077] Where x is the input parameter, f(x) is the output function value, and α is the activation factor, which is usually taken as 0.01. The add module performs numerical addition and fusion operations on the extracted features, and the Concat layer performs channel dimension splicing operations on the two feature layers. The module contains two small residual structures and one large residual structure. The two small residual structures are connected in series to achieve feature reuse. The large residual structure connects the input and output ends of the module, optimizes the repeated gradient information in the network, increases the difference between gradient information, and propagates the gradient flow through different network paths by truncating the gradient flow, thereby improving the learning ability of the algorithm and reducing the amount of model calculation. Stronger network learning ability is beneficial to the extraction of small flame features and is helpful to improving the detection effect of early fireworks targets. This module is embedded in the algorithm structure after the three Concat channel splicing and is applied. The specific location is as follows Figure 5 shown.

[0078] Furthermore, to support the edge device deployment of the algorithm, the present invention uses the Ghost convolution operation to replace the standard convolution operation to reduce the number of algorithm parameters and reduce memory usage. The Ghost convolution processing flow is as follows: Figure 8As shown in the figure, this convolution is a lightweight convolution operation. First, a 1*1 standard convolution is used to obtain a multi-channel feature layer. Then, in order to reduce redundant convolution kernels, Ghost convolution performs linear convolution operations on the obtained multi-channel feature layer channel by channel, and concatenates and stacks the feature layers obtained from the two parts. While ensuring the number of output feature maps, the number of convolution kernels used is greatly reduced.

[0079] The inference process of the fireworks recognition model is as follows Fig. 9 As shown in the figure, the fireworks recognition model receives the predicted image and preprocesses the image. The CSPDarknet backbone feature extraction network is used to perform preliminary feature extraction. The SPPF spatial pyramid pooling structure integrates the initially extracted features to improve the model receptive field. The path aggregation network fuses semantic features with fine-grained features to prevent the loss of detailed information of fireworks targets. The cross-stage residual module optimizes gradient information. The polarized self-attention structure suppresses background interference and enhances the significance of fireworks target information. The prediction information of the model is further output by the detection head and combined with the non-maximum suppression algorithm to obtain the recognition result of the fireworks target.

[0080] The detection results of the fireworks recognition model are as follows: Fig.10 As shown:

[0081] Secondly, refer to the attached Figure 2 A fireworks recognition device according to an embodiment of the present invention is described.

[0082] Figure 2 It is a block diagram of a fireworks recognition device according to an embodiment of the present invention.

[0083] like Figure 2 As shown, the fireworks identification device includes:

[0084] A data preprocessing module is used to obtain fireworks image data and annotate the image data to construct a fireworks data set;

[0085] A data set partitioning module is used to partition the fireworks data set into a training set for building a model, a validation set for model parameter adjustment and optimization, and a test set for evaluating model performance;

[0086] Neural network model building module, used to build a fireworks recognition model based on convolutional neural network;

[0087] Model training and optimization module, used to train the fireworks recognition model using the training set and adjust the hyperparameters of the fireworks recognition model using the validation set;

[0088] The model verification module is used to use the camera equipment to obtain the video stream, apply the trained fireworks recognition model to infer and judge the video stream, and detect whether there is a fire point in the monitored area.

[0089] It should be noted that the above explanations of the fireworks recognition method embodiment are also applicable to the fireworks recognition device of this embodiment, and will not be repeated here.

[0090] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include:

[0091] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .

[0092] When the processor 402 executes the program, the fireworks recognition method provided in the above embodiment is implemented.

[0093] Furthermore, the electronic device further comprises:

[0094] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0095] The memory 401 is used to store computer programs that can be executed on the processor 402 .

[0096] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0097] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0098] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0099] The processor 402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0100] The embodiment of the present invention further provides a computer program product, and when the computer program / instructions are executed by a processor, the above-mentioned fireworks recognition method is implemented.

[0101] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned fireworks recognition method is implemented.

[0102] The fireworks recognition algorithm of the fireworks recognition model is embedded in the edge computing chip, and the trained recognition algorithm model needs to be converted, quantified, and adapted. The edge deployment process of the fireworks recognition algorithm is as follows: Fig.11 As shown in the figure, the specific steps of edge deployment of the fireworks recognition algorithm are as follows:

[0103] 1. Use the ONNX tool to convert the trained fireworks recognition model and build an ONNX calculation graph to describe the calculation process of the model. The ONNX graph consists of nodes, each of which represents an operation. The nodes are connected through input and output to form a directed acyclic graph.

[0104] 2. Map its own operations to the ONNX operation set, and convert its own data types to the data types supported by ONNX to obtain the ONNX format model.

[0105] 3. Use ONNX Runtime or other ONNX-compatible tools to verify that the exported ONNX model is correct.

[0106] 4. The ONNX model is verified to be correct, and the RKNN Toolkit is used to parse the input ONNX model. The parsed model is converted into a computational graph represented internally by the RKNN Toolkit.

[0107] 5. Quantize the model by converting floating-point weights and activation values ​​into fixed-point numbers to reduce model size and computational complexity.

[0108] To improve the running efficiency on the processor, the model is further optimized using memory layout optimization, parallel computing optimization, etc. A model file in RKNN format is generated based on the optimized calculation graph.

[0109] The present invention is a firework recognition system based on a domestically produced edge chip. It uses an infrared lens and a visible light lens to collect monitoring video data, transmits the video stream to the domestically produced edge chip in real time through network communication, performs video stream preprocessing, hardware encoding and decoding, and firework recognition algorithm analysis in the chip, controls the pan-tilt device through serial communication, and displays the analyzed video on the web through the network port. The schematic diagram of the firework recognition system based on the domestically produced edge chip is shown in the figure. Fig.11 shown.

[0110] In order to solve the problems of expensive hardware cost for algorithm deployment and core technology being controlled by others, this paper designs a domestically produced edge chip based on Rockchip RK3588 chip to deploy and apply the fireworks recognition algorithm at the edge, thus reducing the transmission and storage of sensitive data, the risk of data leakage and the time of data transmission to the cloud or central server.

[0111] Edge computing chips such as Fig.12 , Fig.13 As shown, the domestically produced edge chip designed by the present invention is based on the Rockchip RK3588 chip, including 232 serial ports, 485 serial ports, gigabit network ports, NPU, CPU and other modules, and is powered by a 12V power supply. It has the functions of inference acceleration of neural network models, video hardware encoding and decoding, serial communication and network communication. An infrared lens and a visible light lens are used to monitor the collection of video data, and the video stream is transmitted to the domestically produced edge chip in real time through network communication for processing. The video stream is pre-processed, hardware encoded and decoded, and fireworks recognition algorithm analyzed in the chip, and the pan-tilt device is controlled by serial communication, and the analyzed video is displayed on the web through the network port. The schematic diagram of the fireworks recognition system of the domestically produced edge chip is shown in the figure. Fig.14 shown.

[0112] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0113] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0114] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0115] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0116] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0117] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0118] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0119] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for identifying fireworks, characterized in that: The following steps are involved: Obtain fireworks image data and annotate the image data to build a fireworks dataset; The fireworks dataset is divided into a training set for model building, a validation set for model parameter adjustment and optimization, and a test set for evaluating model performance. Use convolutional neural network as the basis to build a fireworks recognition model; Use the training set to train the fireworks recognition model, and use the validation set to adjust the hyperparameters of the fireworks recognition model; Use the camera to obtain the video stream, apply the trained fireworks recognition model to infer and judge the video stream, and detect whether there is a fire point in the monitoring area.

2. A method for identifying fireworks according to claim 1, characterized in that: When constructing the fireworks dataset, mosaic data augmentation is used to expand the fireworks dataset.

3. A method for identifying fireworks according to claim 1, characterized in that: The ratio of the training set, validation set and test set is 8:1:

1.

4. A method for identifying fireworks according to claim 1, 2 or 3, characterized in that: The fireworks recognition model is constructed as follows: the YOLOv8 algorithm is used as the benchmark model, polarized self-attention (PSA) is introduced to weight image features, a cross-stage residual module is designed based on the idea of ​​CSPNet (Cross Stage Partial Networks) and feature reuse, and Ghost convolution processing is used.

5. A method for identifying fireworks according to claim 4, characterized in that: The polarized self-attention includes a channel attention branch and a spatial attention branch. The Softmax function and the Sigmoid function are mixed in the attention calculation of the channel attention branch and the spatial attention branch.

6. A method for identifying fireworks according to claim 4, characterized in that: The Ghost convolution processing flow is as follows: first, a 1*1 standard convolution is used to obtain a multi-channel feature layer, and the Ghost convolution performs a linear convolution operation on the obtained multi-channel feature layer channel by channel, and then splices and stacks them.

7. A method for identifying fireworks according to claim 4, characterized in that: The reasoning and judgment includes the following steps: the fireworks recognition model receives the predicted image and preprocesses the image, performs preliminary feature extraction through the CSPDarknet backbone feature extraction network, integrates the preliminarily extracted features through the SPPF spatial pyramid pooling structure to improve the model receptive field, the path aggregation network fuses the semantic features with the fine-grained features, the cross-stage residual module optimizes the gradient information, the polarized self-attention structure strengthens the significance of the fireworks target information and suppresses background interference, the detection head outputs the predicted information, and the NMS non-maximum suppression outputs the fireworks target recognition result.

8. A fireworks identification device, characterized in that: include: A data preprocessing module is used to obtain fireworks image data and annotate the image data to construct a fireworks data set; A data set partitioning module is used to partition the fireworks data set into a training set for building a model, a validation set for model parameter adjustment and optimization, and a test set for evaluating model performance; Neural network model building module, used to build a fireworks recognition model based on convolutional neural network; Model training and optimization module, used to train the fireworks recognition model using the training set and adjust the hyperparameters of the fireworks recognition model using the validation set; The model verification module is used to use the camera equipment to obtain the video stream, apply the trained fireworks recognition model to infer and judge the video stream, and detect whether there is a fire point in the monitored area.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method according to claim 1, 2, 3, 5, 6 or 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, the computer program being used to cause the computer to execute the method as claimed in claim 1, 2, 3, 5, 6 or 7.

Images