Real-time detection method for smoke and fire easily-confused target

By building a target detector and a confusing classifier, combined with YOLOv8 and MobileNetV3 models, the problem of misdetecting targets of confusing fire smoke in forest fire detection is solved, efficient real-time detection is achieved, reducing labeling costs and improving recognition accuracy .

CN119048894APending Publication Date: 2024-11-29CENT CHINA BRANCH OF STATE GRID CORP OF CHINA +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410059690.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-01-16
Publication Date
2024-11-29

Smart Images

  • Figure CN119048894A_ABST
    Figure CN119048894A_ABST
Patent Text Reader

Abstract

The real-time detection method for the smoke and fire easily-confused target comprises the following steps: 1) constructing a first category target detector and a plurality of easily-confused target classifiers; 2) acquiring a real-time image; 3) inputting the real-time image into a first category target detector to obtain a plurality of target position frames and a corresponding smoke and fire first category; 4) judging whether the first type of smoke and fire belongs to an easily confused target type, if so, entering step 5), and if not, taking the first type of smoke and fire as the type of the current target position frame; 5) selecting a corresponding easy-to-confuse target classifier according to the first type of the smoke and fire; classifying the images in the current target position frame by using the selected easy-to-confuse target classifier to obtain a second type of the smoke and fire as the type of the current target position frame; and 6) outputting all detected target position frames and corresponding categories to the real-time image in the step 2). According to the method, the balance between the precision and the speed is realized, so that the model can achieve a better effect without extra labeling cost.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the field of image detection, and in particular to a real-time detection method for targets easily confused by fireworks. Background Art

[0002] Forest fire is a natural disaster that is sudden, destructive, and difficult to deal with. Forest fire not only kills and burns trees, directly reducing forest area, but also seriously damages forest structure and environment, causing the forest ecosystem to lose balance, forest biomass to decrease, productivity to weaken, beneficial animals and birds to decrease, and even cause casualties of humans and animals. Therefore, it is very necessary to detect forest fire images.

[0003] Image detection of forest fires is generally subjective and manual monitoring, but the forest area is large, and manual monitoring may not be able to detect the location of the fire in time. Intelligent image monitoring may lead to the problem of misdetection of objects that are easily confused with fire and smoke. Therefore, it is necessary to solve the problem of misdetection of objects that are easily confused with fire and smoke (such as fire-like lights and smoke-like fog) in existing forest fire image detection algorithms.

[0004] In order to solve the problem of misdetection of easily confused fire-like and smoke-like targets (such as fire-like lights and smoke-like fog) in existing forest fire image detection algorithms, the existing technology has proposed some target detection algorithms. However, these target detection algorithms still have the following problems: 1) The convolution steps are many and complex and have not been lightweight designed, resulting in slow model inference speed and cannot be applied to real-time tasks, and cannot be applied to forest fire detection tasks that require real-time performance; 2) There is a lot of background noise in the input image of the model, resulting in unsatisfactory test results in actual scenes, and the model has not been lightweight designed, the use of model parameters is not efficient, and the fine classification method will lead to high cost of manual labeling of data sets.

[0005] In summary, the existing methods mainly have two defects: slow reasoning speed or low recognition accuracy. The cost of dataset annotation is also an issue that needs attention. Summary of the invention

[0006] The object of the present invention is to provide a method for real-time detection of targets easily confused by fireworks, comprising the following steps:

[0007] 1) Constructing the first category target detector and multiple confusing target classifiers, including the following steps:

[0008] 1.1) Obtain historical fireworks detection data and construct a first-category training set and multiple second-category training sets.

[0009] The first category training set includes a plurality of positive samples and a plurality of negative samples.

[0010] The positive sample is an image of the first category including fireworks.

[0011] The negative samples are images of the first category that do not contain fireworks.

[0012] The first category training set Q is as follows:

[0013]

[0014] In the formula, vector B1={B 11 ,B 12 ,...,B 1N} represents image data containing fire-like objects. Vector B2 = {B 21 ,B 22 ,...,B 2N} represents image data containing smoke-like objects. Vector B3 = {B 31 ,B 32 ,...,B 3N} represents image data containing mechanical objects. Vector C1 = {C 11 ,C 12 ,...,C 1N} represents the image data containing negative samples. N represents the total number of fireworks first category image data.

[0015] The second category training set includes a fire-like object training set, a smoke-like object training set, and a mechanical object training set.

[0016] The fire-like object training set Q1 is as follows:

[0017] Q1=[D g1 D g2 … D gM ] (2)

[0018] In the formula, vector D g ={D g1 ,D g2 ,...,D gM} represents the image data containing the g-th category of fire objects. g is a positive integer. M represents the total number of second category image data of fireworks.

[0019] The smoke-like object training set Q2 is as follows:

[0020] Q2=[E h1 E h2 … E hM ] (3)

[0021] In the formula, the vector E h ={E h1 ,E h2 ,...,E hM} represents image data containing the h-th type of smoke object. h is a positive integer.

[0022] The mechanical object training set Q3 is as follows:

[0023] Q3=[F r1 F r2 … F rM ] (4)

[0024] In the formula, the vector F r ={F r1 ,F r2 ,...,F rM} represents image data containing the rth mechanical object. r is a positive integer.

[0025] 1.2) Using a neural network to construct a target detector, and using the first category training set to train the target detector, to obtain a first category target detector.

[0026] When training the target detector using the first category training set, the target detector is updated by the overall loss function of the neural network, and the overall loss function λ of the neural network is as follows:

[0027] λ=λ box +λ obj +λ cls (5)

[0028] In the formula, λ box is the CIOU loss function. obj is the BCE loss function.

[0029] Among them, the cross entropy loss function λ cls As shown below:

[0030]

[0031] Where n1 is the batch number, n1=1,2,...,N1, N1 is the batch size, i1 and k1 are both category numbers, and C is the total number of categories. represents the true label. and They all represent the output value of the classification layer.

[0032] The first category object detector is as follows:

[0033] α i =f1(I),(0≤i≤n2) (7)

[0034] In the formula, α i is the first category of the i-th target location box. I is the input image, f1 is the target detector classification process, and n2 is the total number of target location boxes.

[0035] 1.3) Use neural networks to construct multiple groups of easily confused classifiers, and use different types of second category training sets to train these easily confused classifiers respectively, so as to obtain multiple easily confused target classifiers.

[0036] The confusing target classifier is as follows:

[0037] β i =f2(α i ,w i ) (8)

[0038] In the formula, β i is the second category of the i-th target location box. f2 is the confusing classification process.

[0039] Among them, the easily confused target classifier w i As shown below:

[0040] w i =g(α i ) (9)

[0041] Where g is the weight parameter selection function for the easily confused target classification network.

[0042] 2) Get real-time video stream I0 = [I1, I2, ..., I T ]. T Represents the T-th frame of real-time image.

[0043] Among them, the t-th frame real-time image I t As shown below:

[0044]

[0045] In the formula, It represents the pixel value corresponding to the pixel point (e, f) in the t-th frame of the image to be detected, e = 1, 2, ..., m, f = 1, 2, ..., n. m × n represents the size of the real-time image.

[0046] 3) Real-time image I t Perform preprocessing to obtain the image to be detected I t1 , as shown below:

[0047]

[0048] In the formula, Indicates the pixel value corresponding to the pixel point (e, f) in the t-th frame of the image to be detected. v×u represents the size of the image to be detected.

[0049] 4) The image to be detected I t1The first category target detector is input to detect and obtain a plurality of target position frames and the first category of fireworks of each target position frame.

[0050] The target location box is as follows:

[0051]

[0052] In the formula, p 1j 、p 2j 、p 3j 、p 4j Respectively represent the j-th sampling point vector of the first, second, third, and fourth boundary lines of the target position box, j=1,2,...,k, and k is the total number of boundary line sampling points.

[0053] The jth sampling point data vector of the sth boundary line of the target position frame They respectively represent the pixel coordinates and pixel values ​​of the image to be detected corresponding to the j-th sampling point of the s-th boundary line, s=1, 2, 3, 4.

[0054] 5) Traverse the target position frame detected in step 4) to determine whether the first category of fireworks in the current target position frame belongs to the easily confused target category. If so, proceed to step 6); if not, use the first category of fireworks in the current target position frame as the category of the current target position frame.

[0055] 6) According to the first category of fireworks in the current target position frame, select the corresponding easily confused target classifier.

[0056] The image in the current target position frame is classified using the selected confusable target classifier to obtain the second category of fireworks in the current target position frame, and the second category of fireworks in the current target position frame is used as the category of the current target position frame.

[0057] 7) Output all detected target position frames and corresponding categories to the real-time image I in step 2) t , we get the detection result image I containing the target location box and category text annotation t ',Right now:

[0058]

[0059] In the formula, Represents the pixel value corresponding to the pixel point (e, f) of the detection result image in the tth frame.

[0060] Furthermore, the first category of fireworks includes a category of easily confused targets and a category of non-easy confused targets.

[0061] The easily confused target categories include fire-like objects, smoke-like objects, and mechanical objects.

[0062] The non-confusable target categories include pedestrians and buildings.

[0063] Furthermore, the second category of the fire-like objects includes fire, light, and sunset.

[0064] Furthermore, the second category of the smoke-like objects includes smoke, cloud, and fog.

[0065] Furthermore, the second category of the mechanical objects includes excavators, bulldozers, pump trucks, and cranes.

[0066] Furthermore, the preprocessing in step 3) includes random flipping, rotation, cropping, deformation and scaling, random erasing, mosaic, scaling and vector normalization.

[0067] Furthermore, the historical fire and smoke detection data comes from a fire and smoke detection dataset publicly available on the Internet.

[0068] Furthermore, the public fire and smoke detection dataset includes the UFS-Data dataset.

[0069] Furthermore, the historical fireworks detection data comes from a self-built data set of actual scenes.

[0070] Furthermore, the step of constructing the self-built data set of the actual scene includes: recording the fireworks image to be identified in the fireworks sample library, and intercepting the frames therein through the video stream to name and save them.

[0071] The collected images are marked with target annotation boxes and corresponding second category labels to form a self-built dataset of the actual scene.

[0072] The technical effect of the present invention is unquestionable. The present invention takes into account the recognition accuracy and inference speed of the model, achieves a trade-off between accuracy and speed, and solves the problem of increasing the complexity of annotation in order to improve classification accuracy in the related work UFS-Net, so that the model can achieve better results without additional annotation costs.

[0073] In terms of recognition accuracy, the present invention adopts a two-stage architecture of detection-recognition. When facing easily confused targets, the target detector only performs position detection and rough classification, and moves the easily confused fine-grained classification tasks to the downstream image classification model, so that the target detector and image classifier can be trained separately in a targeted manner. Secondly, facing different easily confused large categories (fire-like, smoke-like, machinery-like), the present invention trains different image classifiers, such as fire-like objects such as fire and light sunset, smoke-like objects such as smoke and fog, etc., and smoke-like objects such as excavators, bulldozers, pump trucks, cranes, etc., are trained separately. A multi-classification model is trained separately for multi-classified easily confused large machinery. Its purpose is to use each individually trained easily confused model as a fine-grained classification model, that is, only classify small-class targets with the same large-class features, and the model parameters only contain the common features of a single large class and the specific features of its small class, and will not be confused with other large classes. For example, the downstream classification model parameters of fire and fire-like targets only include the target features of fire and fire-like targets, and do not include the features of other major categories such as smoke and machinery. This can effectively improve the separability of easily confused targets in the same major category, thereby improving the overall recognition accuracy of the model.

[0074] In terms of reasoning speed, the present invention uses the most advanced single-stage target detector YOLOv8 in the target detection part, which can achieve single-stage target detection on the COCO dataset. Figure 2 The inference speed is about ms, and for a 640*640 wildfire detection image, it can also reach an inference speed of about 15ms. In the image classification part, a lightweight neural network MobileNetV3 is used, which uses deep separable convolution to split a standard convolution into deep convolution and point convolution, which greatly improves the computing efficiency while basically maintaining the original accuracy. For a 224*224 classified image, the inference speed can reach less than 10ms. In summary, the present invention adopts a lightweight design, uses a lightweight target detector and image classifier, and optimizes the recognition process: for detection images with a resolution higher than 640, they are first compressed, and after input into the detection model, it is judged whether the output classification of the detection target is an easily confused target. If it is an easily confused target, it is input into the downstream classification model for classification; if it is not an easily confused target, the output of the detection model is directly used as the classification result, avoiding the input of the downstream image classification model and improving the inference speed.

[0075] In terms of data set annotation, the categories that need to be annotated in this invention mainly include wildfire positive samples and fire-like negative samples, smoke positive samples and smoke-like negative samples, and different types of large machinery. By reasonably designing positive and negative samples, the recognition accuracy is improved and the cost of manual annotation is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is the overall flow chart of the present invention;

[0077] Figure 2 This is the overall architecture diagram of the algorithm of the present invention. DETAILED DESCRIPTION

[0078] The present invention is further described below in conjunction with the embodiments, but it should not be understood that the above subject matter of the present invention is limited to the following embodiments. Without departing from the above technical ideas of the present invention, various substitutions and changes are made according to the common technical knowledge and customary means in the art, which should all be included in the protection scope of the present invention.

[0079] Embodiment 1:

[0080] See also Figures 1 to 2 ,A real-time detection method for targets easily confused by fireworks includes the following steps:

[0081] 1) Constructing the first category target detector and multiple confusing target classifiers, including the following steps:

[0082] 1.1) Obtain historical fireworks detection data and construct a first-category training set and multiple second-category training sets.

[0083] The first category training set includes a plurality of positive samples and a plurality of negative samples.

[0084] The positive sample is an image of the first category including fireworks.

[0085] The negative samples are images of the first category that do not contain fireworks.

[0086] The first category training set Q is as follows:

[0087]

[0088] In the formula, vector B1={B 11 ,B 12 ,...,B 1N} represents image data containing fire-like objects. Vector B2 = {B 21 ,B 22 ,...,B 2N} represents image data containing smoke-like objects. Vector B3 = {B 31 ,B 32 ,...,B 3N} represents image data containing mechanical objects. Vector C1 = {C 11 ,C 12 ,...,C 1N} represents the image data containing negative samples. N represents the total number of fireworks first category image data.

[0089] The second category training set includes a fire-like object training set, a smoke-like object training set, and a mechanical object training set.

[0090] The fire-like object training set Q1 is as follows:

[0091] Q1=[D g1 D g2 … D gM ] (2)

[0092] In the formula, vector D g ={D g1 ,D g2 ,...,D gM} represents the image data containing the g-th category of fire objects. g is a positive integer. M represents the total number of second category image data of fireworks.

[0093] The smoke-like object training set Q2 is as follows:

[0094] Q2=[E h1 E h2 … E hM ] (3)

[0095] In the formula, the vector E h ={E h1 ,E h2 ,...,E hM} represents image data containing the h-th type of smoke object. h is a positive integer.

[0096] The mechanical object training set Q3 is as follows:

[0097] Q3=[F r1 F r2 … F rM ] (4)

[0098] In the formula, the vector F r ={F r1 ,F r2 ,...,F rM} represents image data containing the rth mechanical object. r is a positive integer.

[0099] 1.2) Using a neural network to construct a target detector, and using the first category training set to train the target detector, to obtain a first category target detector.

[0100] When training the target detector using the first category training set, the target detector is updated by the overall loss function of the neural network, and the overall loss function λ of the neural network is as follows:

[0101] λ=λ box +λ obj +λ cls (5)

[0102] In the formula, λbox is the CIOU loss function. obj is the BCE loss function.

[0103] Among them, the cross entropy loss function λ cls As shown below:

[0104]

[0105] Where n1 is the batch number, n1=1,2,...,N1, N1 is the batch size, i1 and k1 are both category numbers, and C is the total number of categories. represents the true label. and They all represent the output value of the classification layer.

[0106] The first category object detector is as follows:

[0107] α i =f1(I),(0≤i≤n2) (7)

[0108] In the formula, α i is the first category of the i-th target location box. I is the input image, f1 is the target detector classification process, and n2 is the total number of target location boxes.

[0109] 1.3) Use neural networks to construct multiple groups of easily confused classifiers, and use different types of second category training sets to train these easily confused classifiers respectively, so as to obtain multiple easily confused target classifiers.

[0110] The confusing target classifier is as follows:

[0111] β i =f2(α i ,w i ) (8)

[0112] In the formula, β i is the second category of the i-th target location box. f2 is the confusing classification process.

[0113] Among them, the easily confused target classifier w i As shown below:

[0114] w i =g(α i ) (9)

[0115] Where g is the weight parameter selection function for the easily confused target classification network.

[0116] 2) Get real-time video stream I0 = [I1, I2, ..., I T ]. T Represents the T-th frame of real-time image.

[0117] Among them, the t-th frame real-time image I t As shown below:

[0118]

[0119] In the formula, It represents the pixel value corresponding to the pixel point (e, f) in the t-th frame of the image to be detected, e = 1, 2, ..., m, f = 1, 2, ..., n. m × n represents the size of the real-time image.

[0120] 3) Real-time image I t Perform preprocessing to obtain the image to be detected I t1 , as shown below:

[0121]

[0122] In the formula, Indicates the pixel value corresponding to the pixel point (e, f) in the t-th frame of the image to be detected. v×u represents the size of the image to be detected.

[0123] 4) The image to be detected I t1 The first category target detector is input to detect and obtain a plurality of target position frames and the first category of fireworks of each target position frame.

[0124] The target location box is as follows:

[0125]

[0126] In the formula, p 1j 、p 2j 、p 3j 、p 4j Respectively represent the j-th sampling point vector of the first, second, third, and fourth boundary lines of the target position box, j=1,2,...,k, and k is the total number of boundary line sampling points.

[0127] The jth sampling point data vector of the sth boundary line of the target position frame They respectively represent the pixel coordinates and pixel values ​​of the image to be detected corresponding to the j-th sampling point of the s-th boundary line, s=1, 2, 3, 4.

[0128] 5) Traverse the target position frame detected in step 4) to determine whether the first category of fireworks in the current target position frame belongs to the easily confused target category. If so, proceed to step 6); if not, use the first category of fireworks in the current target position frame as the category of the current target position frame.

[0129] 6) According to the first category of fireworks in the current target position frame, select the corresponding easily confused target classifier.

[0130] The image in the current target position frame is classified using the selected confusable target classifier to obtain the second category of fireworks in the current target position frame, and the second category of fireworks in the current target position frame is used as the category of the current target position frame.

[0131] 7) Output all detected target position frames and corresponding categories to the real-time image I in step 2) t , we get the detection result image I containing the target location box and category text annotation t ',Right now:

[0132]

[0133] In the formula, Represents the pixel value corresponding to the pixel point (e, f) of the detection result image in the tth frame.

[0134] Embodiment 2:

[0135] A method for real-time detection of easily confused targets by fireworks, the main technical content of which is shown in Example 1. Furthermore, the first category of fireworks includes an easily confused target category and a non-easy confused target category.

[0136] The easily confused target categories include fire-like objects, smoke-like objects, and mechanical objects.

[0137] The non-confusable target categories include pedestrians and buildings.

[0138] Embodiment 3:

[0139] A real-time detection method for targets easily confused by fireworks, the main technical content of which is shown in any one of Examples 1 to 2. Furthermore, the second category of the fire-like objects includes fire, light, and sunset.

[0140] Embodiment 4:

[0141] A real-time detection method for targets easily confused by fireworks, the main technical content of which is shown in any one of Examples 1 to 3. Furthermore, the second category of the smoke-like objects includes smoke, clouds, and fog.

[0142] Embodiment 5:

[0143] A real-time detection method for targets easily confused by fireworks, the main technical content of which can be found in any one of Examples 1 to 4. Furthermore, the second category of mechanical objects includes excavators, bulldozers, pump trucks, and cranes.

[0144] Embodiment 6:

[0145] A real-time detection method for targets easily confused by fireworks, the main technical content of which is shown in any one of Examples 1 to 5. Furthermore, the preprocessing in step 3) includes random flipping processing, rotation processing, cropping processing, deformation and scaling processing, random erasing processing, mosaic processing, scaling processing and vector normalization processing.

[0146] Embodiment 7:

[0147] A real-time detection method for targets easily confused by fireworks, the main technical content of which is shown in any one of Examples 1 to 6. Furthermore, the historical fireworks detection data comes from a fireworks detection data set publicly available on the Internet.

[0148] Embodiment 8:

[0149] A real-time detection method for targets easily confused by fireworks, the main technical content of which is shown in any one of Examples 1 to 7. Furthermore, the disclosed fireworks detection dataset includes a UFS-Data dataset.

[0150] Embodiment 9:

[0151] A real-time detection method for targets easily confused by fireworks, the main technical content of which is shown in any one of Examples 1 to 8. Furthermore, the historical fireworks detection data comes from a self-built data set of actual scenes.

[0152] Embodiment 10:

[0153] A method for real-time detection of targets easily confused by fireworks, the main technical content of which is shown in any one of Examples 1 to 9. Furthermore, the step of constructing a self-built data set of the actual scene includes: entering the fireworks image to be identified in the fireworks sample library, and capturing frames therein through the video stream for naming and saving.

[0154] The collected images are marked with target annotation boxes and corresponding second category labels to form a self-built dataset of the actual scene.

[0155] Embodiment 11:

[0156] See also Figures 1 to 2 ,A real-time detection method for targets easily confused by fireworks includes the following steps:

[0157] 1) Constructing the first category target detector and multiple confusing target classifiers, including the following steps:

[0158] 1.1) Obtain historical fireworks detection data and construct a first-category training set and multiple second-category training sets.

[0159] The first category training set includes a plurality of positive samples and a plurality of negative samples.

[0160] The positive sample is an image of the first category including fireworks.

[0161] The negative samples are images of the first category that do not contain fireworks.

[0162] The first category training set Q is as follows:

[0163]

[0164] In the formula, vector B1={B 11 ,B 12 ,...,B 1N} represents image data containing fire-like objects. Vector B2 = {B 21 ,B 22 ,...,B 2N} represents image data containing smoke-like objects. Vector B3 = {B 31 ,B 32 ,...,B 3N} represents image data containing mechanical objects. Vector C1 = {C 11 ,C 12 ,...,C 1N} represents the image data containing negative samples. N represents the total number of fireworks first category image data.

[0165] The second category training set includes a fire-like object training set, a smoke-like object training set, and a mechanical object training set.

[0166] The fire-like object training set Q1 is as follows:

[0167]

[0168] In the formula, vector D1={D 11 ,D 12 ,...,D 1M} represents image data containing fire. Vector D2 = {D 21 ,D 22 ,...,D 2M} represents the image data containing the light. Vector D3 = {D 31 ,D 32 ,...,D 3M} represents image data containing sunset. M represents the total number of second category image data of fireworks.

[0169] The smoke-like object training set Q2 is as follows:

[0170]

[0171] In the formula, vector E1={E 11 ,E 12 ,...,E 1M} represents image data containing smoke. Vector E2 = {E 21,E 22 ,...,E 2M} represents image data containing clouds and fog. Vector E3 = {E 31 ,E 32 ,...,E 3M} indicates image data containing fog.

[0172] The mechanical object training set Q3 is as follows:

[0173]

[0174] In the formula, vector F1={F 11 ,F 12 ,...,F 1M} represents the image data containing the excavator. Vector F2 = {F 21 ,F 22 ,...,F 2M} represents image data containing a bulldozer. Vector F3 = {F 31 ,F 32 ,...,F 3M} represents the image data including the pump truck. Vector F4 = {F 41 ,F 42 ,...,F 4M} indicates that the image data includes the crane.

[0175] The data in the training set includes the location annotation box and the corresponding category of each target.

[0176] 1.2) Use the neural network to build the target detector YOLOv7, and use the first category training set to train the target detector to obtain the first category target detector.

[0177] When training the target detector using the first category training set, the target detector is updated through the overall loss function of the neural network, and the neural network is used for target detection and classification. The network weights are updated during the training process using the loss function to improve the model effect. Among them, the overall loss function of the neural network consists of position coordinate regression loss, target confidence loss and category confidence loss. The overall loss function λ of the neural network is as follows:

[0178] λ=λ box +λ obj +λ cls (5)

[0179] In the formula, λ box is the distance intersection over union loss function (Complete Intersection over Union, CIOU), which is calculated using the real box and the predicted box. objIt is a binary cross entropy (BCE) loss function, which is calculated based on the prediction results of whether the positive and negative samples and the grid feature points contain objects.

[0180] Among them, the cross entropy loss function λ cls The cross entropy loss of the predicted category normalized vector after the softmax activation function is obtained as follows:

[0181]

[0182] Where n1 is the batch number, n1=1,2,...,N1, N1 is the batch size, i1 and k1 are both category numbers, and C is the total number of categories. represents the true label. and They all represent the output value of the classification layer.

[0183] The first category object detector is as follows:

[0184] α i =f1(I),(0≤i≤n2) (7)

[0185] In the formula, α i is the first category of the i-th target location box. I is the input image, f1 is the target detector classification process, and n2 is the total number of target location boxes.

[0186] When training the object detector, all first categories, namely, fire-like, smoke-like, and mechanical, are trained together with the annotation boxes.

[0187] 1.3) Use the MobileNetV3 neural network to build multiple sets of easily confused classifiers, whose classification algorithm is the same as that in the first category target detector. And use different types of second category training sets to train these easily confused classifiers respectively, and obtain multiple easily confused target classifiers.

[0188] The confusing target classifier is as follows:

[0189] β i =f2(α i ,w i ) (8)

[0190] In the formula, β i is the second category of the i-th target location box. f2 is the confusing classification process.

[0191] Among them, the easily confused target classifier w i As shown below:

[0192] w i =g(α i) (9)

[0193] Where g is the weight parameter selection function for the easily confused target classification network.

[0194] When training the confusable classifier, the target in the annotation box is cut out and the target category is trained using the fine-grained second category.

[0195] 2) The local camera pushes the real-time video stream to the cloud server and obtains the real-time video stream I0 = [I1, I2, ..., I T ]. T Represents the T-th frame of real-time image.

[0196] Among them, the t-th frame real-time image I t As shown below:

[0197]

[0198] In the formula, It represents the pixel value corresponding to the pixel point (e, f) in the t-th frame of the image to be detected, e = 1, 2, ..., m, f = 1, 2, ..., n. m × n represents the size of the real-time image.

[0199] 3) Real-time image I t Perform preprocessing to obtain the image to be detected I t1 , as shown below:

[0200]

[0201] In the formula, represents the pixel value corresponding to the pixel point (e, f) in the t-th frame of the image to be detected. v×u represents the size of the image to be detected. Due to image preprocessing, the size of the image to be detected may change.

[0202] 4) The image to be detected I t1 The first category target detector is input to detect and obtain a plurality of target position frames and the first category of fireworks of each target position frame.

[0203] The target location box is as follows:

[0204]

[0205] In the formula, p 1j 、p 2j 、p 3j 、p 4j Respectively represent the j-th sampling point vector of the first, second, third, and fourth boundary lines of the target position box, j=1,2,...,k, and k is the total number of boundary line sampling points.

[0206] The jth sampling point data vector of the sth boundary line of the target position frame They respectively represent the pixel coordinates and pixel values ​​of the image to be detected corresponding to the j-th sampling point of the s-th boundary line, s=1, 2, 3, 4.

[0207] 5) Traverse the target position frame detected in step 4) to determine whether the first category of fireworks in the current target position frame belongs to the easily confused target category. If so, proceed to step 6); if not, use the first category of fireworks in the current target position frame as the category of the current target position frame.

[0208] 6) According to the first category of fireworks in the current target position frame, select the corresponding easily confused target classifier.

[0209] The image in the current target position frame is classified using the selected confusable target classifier to obtain the second category of fireworks in the current target position frame, and the second category of fireworks in the current target position frame is used as the category of the current target position frame.

[0210] 7) Output all detected target position frames and corresponding categories to the real-time image I in step 2) t , we get the detection result image I containing the target location box and category text annotation t ',Right now:

[0211]

[0212] In the formula, Represents the pixel value corresponding to the pixel point (e, f) of the detection result image in the tth frame.

[0213] If a target position frame exists at a pixel point of the detection result image, the pixel value of the detection result image is a superposition of the pixel value of the real-time image and the pixel value of the target position frame.

[0214] If there is no pixel point in the target position box, the pixel value of the detection result image is equal to the pixel value of the real-time image.

[0215] Let Cconfusing be the easily confused category, then steps 5), 6), and 7) can be expressed as follows:

[0216]

[0217] Embodiment 12:

[0218] A real-time detection method for easily confused targets with fireworks, the main technical content of which is shown in Example 11. Furthermore, the first category of fireworks includes an easily confused target category and a non-easy confused target category.

[0219] The easily confused target categories include fire-like objects, smoke-like objects, and mechanical objects.

[0220] The non-confusable target categories include pedestrians, buildings and other objects whose features are obvious and easy to distinguish and do not require further fine-grained classification.

[0221] Embodiment 13:

[0222] A real-time detection method for targets easily confused by fireworks, the main technical content of which can be found in any one of Examples 11 to 12. Furthermore, the second category of the fire-like objects includes fire, light, and sunset.

[0223] Embodiment 14:

[0224] A real-time detection method for targets easily confused by fireworks, the main technical content of which is shown in any one of Examples 11 to 13. Furthermore, the second category of the smoke-like objects includes smoke, clouds, and fog.

[0225] Embodiment 15:

[0226] A real-time detection method for targets easily confused by fireworks, the main technical content of which can be found in any one of Examples 11 to 14. Furthermore, the second category of mechanical objects includes excavators, bulldozers, pump trucks, and cranes.

[0227] The application scenario of this solution is forest fire detection along high-voltage lines. Mechanical objects are prone to cause dangerous scenarios such as electric shock from high-voltage lines. Different mechanical objects correspond to different dangerous scenarios, and different mechanical objects are easily confused with each other, so different mechanical objects need to be identified.

[0228] Embodiment 16:

[0229] A real-time detection method for targets easily confused by fireworks, the main technical content of which can be found in any one of Examples 11 to 15. Furthermore, the preprocessing in step 3) includes random flipping processing, rotation processing, cropping processing, deformation and scaling processing, random erasing processing, mosaic processing, scaling processing and vector normalization processing.

[0230] During the training phase, data enhancement operations such as random flipping, rotation, cropping, deformation and scaling, random erasing, and mosaic are performed to improve the generalization ability of the model; during the testing phase, only scaling and vector normalization operations are performed.

[0231] Embodiment 17:

[0232] A real-time detection method for targets easily confused by fireworks, the main technical content of which is shown in any one of Examples 11 to 16. Furthermore, the historical fireworks detection data comes from a fireworks detection data set publicly available on the Internet.

[0233] In the training phase, the target detector and multiple groups of classifiers need to be trained separately. The target detector can be trained using public fireworks detection datasets such as UFS-Data.

[0234] Embodiment 18:

[0235] A real-time detection method for targets easily confused by fireworks, the main technical content of which is shown in any one of Examples 11 to 17. Furthermore, the disclosed fireworks detection dataset includes a UFS-Data dataset.

[0236] Embodiment 19:

[0237] A real-time detection method for targets easily confused by fireworks, the main technical content of which is shown in any one of Examples 11 to 18. Furthermore, the historical fireworks detection data comes from a self-built data set of actual scenes.

[0238] Embodiment 20:

[0239] A real-time detection method for targets easily confused by fireworks, the main technical content of which can be found in any one of Examples 11 to 19. Furthermore, the step of constructing a self-built data set of the actual scene includes: entering the fireworks image to be identified into the fireworks sample library, and capturing frames therein through the video stream for naming and saving.

[0240] The collected images are marked with target annotation boxes and corresponding second category labels to form a self-built dataset of the actual scene.

[0241] Object detectors can also be trained based on pre-trained models using self-built datasets taken from actual project scenarios.

[0242] Embodiment 21:

[0243] See also Figure 1 ,The real-time detection method of fireworks easily confused targets mainly includes the following contents:

[0244] like Figure 1 As shown, it is the overall flow chart of the present invention, and the specific steps are as follows:

[0245] 1) The local camera pushes real-time video stream to the cloud server, and the server inputs each frame of image into the model after image preprocessing.

[0246] 2) The object detection module performs object detection on the input image.

[0247] 3) Make a judgment based on the classification result of each detected target. If it is an easily confused target, go to step 4); if it is not an easily confused target, go to step 5).

[0248] 4) Confusing targets: Different confusing classifiers are applied according to the target detection category, and the results are output as categories.

[0249] 5) Non-confusable targets directly use the classification of the target detector as the category output.

[0250] The category obtained in step 4) or 5) and the target position box output by the target detector are finally output.

[0251] Among them, the target detector uses the most advanced YOLOv8, and the classifier for easily confused targets uses the lightweight neural network MobileNetV3. In the training phase, multiple sets of different classification weight parameters are trained according to the actual application scenarios.

[0252] During the training phase, the target detector and multiple groups of classifiers need to be trained separately. The target detector can be trained using public fire and smoke detection datasets such as UFS-Data, or it can be trained using self-built datasets taken from actual project scenarios based on pre-trained models. The number of classifier groups depends on the actual task requirements. For example, in this fire and smoke detection task, fire and light and other fire-like objects are a group of easily confused objects, smoke and fog and other smoke-like objects are a group of easily confused objects, and large machinery such as excavators, bulldozers, pump trucks, cranes, etc. are a group of easily confused objects, so 3 different groups of classifiers should be trained for the above categories.

[0253] Embodiment 22:

[0254] See also Figure 2 ,The real-time detection method of fireworks easily confused targets mainly includes the following contents:

[0255] like Figure 2 As shown in FIG. 1 , it is a diagram of the overall algorithm architecture of the present invention. In the test phase, an image containing flames is input, and the specific algorithm execution steps are as follows:

[0256] 1) After image preprocessing, the model input image is obtained.

[0257] 2) Input the image into the object detector YOLOv8 and obtain the flame position and its category.

[0258] 3) According to the target classification results of YOLOv8, flame is an easily confused target and belongs to the first category of the easily confused fire group.

[0259] 4) Select the fire-like classification weights from the trained MobileNetV3 weight library and load it into the MobileNetV3 model.

[0260] 5) Use MobileNetV3 loaded with specific weights to classify the detected flames and obtain the final result.

Claims

1. A real-time detection method for targets easily confused by fireworks, characterized in that: The following steps are involved: 1) Constructing the first category target detector and multiple confusing target classifiers, including the following steps: 1.1) Obtain historical fireworks detection data and construct a first-category training set and multiple second-category training sets; The first category training set includes a plurality of positive samples and a plurality of negative samples. The positive sample is an image of the first category including fireworks. The negative sample is an image of the first category that does not contain fireworks; The first category training set Q is as follows: In the formula, vector B1={B 11 ,B 12 ,...,B 1N } represents image data containing fire-like objects; vector B2 = {B 21 ,B 22 ,...,B 2N } represents image data containing smoke-like objects; vector B3 = {B 31 ,B 32 ,...,B 3N } represents image data containing mechanical objects; vector C1 = {C 11 ,C 12 ,...,C 1N } represents the image data containing negative samples; N represents the total number of fireworks first category image data; The second category training set includes a fire-like object training set, a smoke-like object training set, and a mechanical object training set; The fire-like object training set Q1 is as follows: Q1=[D g1 D g2 … D gM ] (2) In the formula, vector D g ={D g1 ,D g2 ,...,D gM } represents the image data containing the g-th category of fire objects; g is a positive integer; M represents the total number of the second category of fireworks image data; The smoke-like object training set Q2 is as follows: Q2=[E h1 E h2 … E hM ] (3) In the formula, the vector E h ={E h1 ,E h2 ,...,E hM } represents image data containing the hth type of smoke object; h is a positive integer; The mechanical object training set Q3 is as follows: Q3=[F r1 F r2 …F rM ] (4) In the formula, the vector F r ={F r1 ,F r2 ,...,F rM } represents image data containing the rth mechanical object; r is a positive integer; 1.2) constructing a target detector using a neural network, and training the target detector using the first category training set to obtain a first category target detector; When training the target detector using the first category training set, the target detector is updated by the overall loss function of the neural network, and the overall loss function λ of the neural network is as follows: λ=λ box +λ obj +λ cls (5) In the formula, λ box is the CIOU loss function; obj is the BCE loss function; Among them, the cross entropy loss function λ cls As shown below: Where n1 is the batch number, n1=1,2,...,N1, N1 is the batch size; i1 and k1 are both category numbers, C is the total number of categories; represents the true label; and All represent the output value of the classification layer; The first category object detector is as follows: α i =f1(I),(0≤i≤n2) (7) In the formula, α i is the first category of the i-th target location box; I is the input image, f1 is the target detector classification process, and n2 is the total number of target location boxes; 1.3) Using a neural network to construct multiple sets of easily confused classifiers, and using different types of second category training sets to train these easily confused classifiers respectively, to obtain multiple easily confused target classifiers; The confusing target classifier is as follows: β i =f2(α i ,w i ) (8) In the formula, β i is the second category of the i-th target location box; f2 is the confusing classification process; Among them, the easily confused target classifier w i As shown below: w i =g(α i ) (9) Where g is the weight parameter selection function of the easily confused target classification network; 2) Get real-time video stream I0 = [I1, I2, ..., I T ];I T Represents the real-time image of frame T; Among them, the t-th frame real-time image I t As shown below: In the formula, represents the pixel value corresponding to the pixel point (e, f) in the t-th frame of the image to be detected, e=1,2,...,m, f=1,2,...,n; m×n represents the size of the real-time image; 3) Real-time image I t Perform preprocessing to obtain the image to be detected I t1 , as shown below: In the formula, represents the pixel value corresponding to the pixel point (e, f) in the t-th frame of the image to be detected; v×u represents the size of the image to be detected; 4) The image to be detected I t1 Input a first category target detector to detect and obtain a plurality of target position frames and a first category of fireworks in each target position frame; The target location box is as follows: In the formula, p 1j 、p 2j 、p 3j 、p 4j Respectively represent the j-th sampling point vector of the first, second, third, and fourth boundary lines of the target position box, j = 1, 2, ..., k, k is the total number of boundary line sampling points; The jth sampling point data vector of the sth boundary line of the target position frame Respectively represent the coordinates and pixel values ​​of the image pixel to be detected corresponding to the j-th sampling point of the s-th boundary line, s = 1, 2, 3, 4; 5) Traverse the target position frame detected in step 4) to determine whether the first category of fireworks in the current target position frame belongs to the easily confused target category. If so, proceed to step 6). If not, take the first category of fireworks in the current target position frame as the category of the current target position frame; 6) According to the first category of fireworks in the current target position frame, select the corresponding easily confused target classifier; and using the selected confusable target classifier to classify the image in the current target position frame to obtain the second category of fireworks in the current target position frame, and using the second category of fireworks in the current target position frame as the category of the current target position frame; 7) Output all detected target position frames and corresponding categories to the real-time image I in step 2) t , we get the detection result image I containing the target location box and category text annotation t ',Right now: In the formula, Represents the pixel value corresponding to the pixel point (e, f) of the detection result image in the tth frame.

2. The real-time detection method for targets easily confused by fireworks according to claim 1 is characterized in that: The first category of fireworks includes a confusing target category and a non-confusing target category; The easily confused target categories include fire-like objects, smoke-like objects, and mechanical objects; The non-confusable target categories include pedestrians and buildings.

3. The real-time detection method for targets easily confused by fireworks according to claim 2 is characterized in that: The second category of the fire-like objects includes fire, light, and sunset.

4. The method for real-time detection of targets easily confused by fireworks according to claim 3, characterized in that: The second category of the smoke-like objects includes smoke, cloud, and fog.

5. The method for real-time detection of targets easily confused by fireworks according to claim 4, characterized in that: The second category of the mechanical objects includes excavators, bulldozers, pump trucks, and cranes.

6. The real-time detection method for targets easily confused by fireworks according to claim 1, characterized in that: The preprocessing in step 3) includes random flipping, rotation, cropping, deformation and scaling, random erasing, mosaic, scaling and vector normalization.

7. The real-time detection method for targets easily confused by fireworks according to claim 1 is characterized in that: The historical fire and smoke detection data comes from a fire and smoke detection dataset publicly available on the Internet.

8. The method for real-time detection of targets easily confused by fireworks according to claim 7, characterized in that: The public fire and smoke detection dataset includes the UFS-Data dataset.

9. The method for real-time detection of targets easily confused by fireworks according to claim 1, characterized in that: The historical fireworks detection data comes from a self-built data set of actual scenes.

10. The method for real-time detection of targets easily confused by fireworks according to claim 9, characterized in that: The steps of constructing the self-built data set of the actual scene include: recording the fireworks image to be identified in the fireworks sample library, intercepting the frames therein through the video stream, naming and saving them; The collected images are marked with target annotation boxes and corresponding second category labels to form a self-built dataset of the actual scene.

Citation Information

Cited By

  • Biological diversity intelligent monitoring method and system based on cloud platform

    CN120726307A

  • Cloud-based intelligent biodiversity monitoring methods and systems

    CN120726307B