Construction site fire monitoring method based on deep learning and multi-source image fusion perception

Abstract

The application discloses a construction site fire monitoring method based on deep learning and multi-source image fusion perception, and comprises the following steps: S1, target detection networks under visible light and infrared light are respectively established; S2, construction site operation videos are respectively input into the visible light target detection network and the infrared light target detection network, and the boundary box parameters of fire and person are respectively obtained; S3, the result difference of the visible light target detection network and the infrared light target detection network is calculated, compared with a threshold value, and the detection results are fused; S4, whether fire and person exist is determined, and the relative positions of the fire and the person are recorded. The method can automatically identify the fire operation state, effectively avoid the defect that single image source detection is inaccurate due to external environment changes, improve the accuracy of the fire state identification, reduce the false alarm rate, effectively reduce the interference of the sensor with poor detection effect on the detection result, and improve the construction site safety level and management efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of construction site fire monitoring, and in particular to a construction site fire monitoring method based on deep learning and multi-source image fusion perception. BACKGROUND

[0002] At present, there are many construction sites, many construction equipment and many construction personnel in the field of construction. Due to the complex production operation environment of the construction site, the complex personnel, the scattered construction site and the high difficulty of the construction site management, combined with the management loopholes, safety accidents occur every year. The relevant departments of the state pay great attention to the safety of the construction site, and issue a series of policy documents to ensure the safety of the construction site, and combine the advanced information technology to improve the intelligent degree of the construction site management.

[0003] However, in the current construction site management system, the progress and quality of the project, the safety of the personnel and the environment are monitored and managed, and the monitoring and management of the fire operation are relatively insufficient. As one of the frequent operations in engineering construction, the fire operation has a certain risk, and the fire often causes personal injury or significant property loss. At present, single visible light detection is used to determine whether there is an open fire, but single visible light detection may not correctly identify the working state of the fire operation. Therefore, it is necessary to develop a construction site fire monitoring method. SUMMARY

[0004] In view of the problems in the prior art, the present application provides a construction site fire monitoring method based on deep learning and multi-source image fusion perception.

[0005] To solve the above technical problems, the present application includes the following technical solutions:

[0006] A construction site fire monitoring method based on deep learning and multi-source image fusion perception includes the following steps:

[0007] S1. Establish target detection networks under visible light and infrared light respectively, train the target detection networks with fire operation images, and obtain the required visible light target detection network and infrared light target detection network;

[0008] S2. Use a video acquisition device to acquire a construction site operation video, and input the construction site operation video into the visible light target detection network and the infrared light target detection network respectively, identify the distribution of the fire and the person in each frame of image, and obtain the boundary box parameters of the fire and the person respectively;

[0009] S3. According to the boundary box parameters of the target area obtained in step S2, calculate the difference between the results of the visible light target detection network and the infrared light target detection network, compare with the threshold value, and fuse the detection results;

[0010] S4. According to the fusion detection result of step S3, it is determined whether there is fire and personnel, and the relative position of the fire and personnel is recorded.

[0011] Further, step S1 trains the target detection network using the fire operation image to obtain a target detection network meeting the requirements, and specifically includes the following steps:

[0012] S11. A sample database is established, containing a plurality of fire operation images in a work site;

[0013] S12. The gap images in the sample database are randomly divided into a training set and a test set according to a predetermined proportion, and the boundary box parameters and the class of the target region of the fire operation images in the training set are labeled, wherein the class includes fire and personnel;

[0014] S13. The target detection network under visible light and infrared light is used to iteratively train the fire operation images in the training set, and the structure and parameters of the network are continuously adjusted until the training parameters meet the requirements, and a trained target detection network is obtained.

[0015] Further, the target detection network adopts a YOLOv3 network.

[0016] Further, the boundary box parameters of the fire and the personnel obtained in step S2 include four-dimensional position parameters and one-dimensional confidence parameters .

[0017] wherein, wherein ,, and respectively represent the x coordinate of the left boundary, the x coordinate of the right boundary, the y coordinate of the upper boundary, and the y coordinate of the lower boundary of the target region boundary box, wherein the coordinate origin is located at the upper left corner of the input monitoring image.

[0018] Further, in step S3, the difference between the results of the visible light target detection network and the infrared light target detection network is calculated, compared with a threshold value, and the detection results are fused, and specifically includes the following steps:

[0019] S31. Boundary box center point calculation: according to the boundary box parameters of the detection region under visible light and infrared light obtained in step S2 , the center point coordinates and the area of each boundary box are calculated, and the calculation formula is as follows:

[0020] ;

[0021] ;

[0022] ;

[0023] ;

[0024] ;

[0025] ;

[0026] wherein, and respectively represent the center point coordinates of the fire and personnel bounding boxes detected by the target detection network under visible light and infrared light; is the four-dimensional position parameter of the bounding box under visible light, is the four-dimensional position parameter of the bounding box under infrared light; and are the areas of the fire and personnel bounding boxes detected by the target detection network under visible light and infrared light, respectively;

[0027] S32. Bounding box area ratio calculation: according to the areas of the fire and personnel bounding boxes detected by the target detection network under visible light and infrared light, calculate the proportion of the weighted area sum, the calculation formula is as follows:

[0028] ;

[0029] ;

[0030] wherein and respectively represent the proportion of the areas of the fire and personnel bounding boxes detected by the target detection network under visible light and infrared light to the weighted area sum; represents the threshold value of the area ratio; is the penalty coefficient;

[0031] S33. Fusion bounding box detection result calculation: according to the area ratio obtained in S32, fuse the target detection results under visible light and infrared light to obtain the predicted fire and personnel bounding boxes, the calculation formula is as follows:

[0032] ;

[0033] ;

[0034] ;

[0035] ;

[0036] wherein and These represent the center coordinates of the bounding box of the fused detection result. and These represent the width and height of the bounding box of the fused detection result, respectively.

[0037] By employing the above technical solutions, this invention has the following advantages and positive effects compared with the prior art:

[0038] The hot work monitoring method for construction sites based on deep learning and multi-source image fusion perception provided by this invention uses a target detection network established by deep learning to identify images captured by construction site monitoring videos, thereby monitoring the status of hot work operations. This eliminates the need for manual monitoring, freeing up manpower and making it highly practical.

[0039] The present invention provides a construction site hot work monitoring method based on deep learning and multi-source image fusion perception. Based on the detection materials of visible light and infrared light sensors, two target detection networks are used to detect the fire and person regions in the image, respectively, and the area of ​​the detection box is calculated for each region. By fusing the detection results, the method can effectively avoid the defects of inaccurate detection from a single image source caused by changes in the external environment, improve the accuracy of hot work status recognition, and reduce the false alarm rate.

[0040] The construction site hot work monitoring method based on deep learning and multi-source image fusion perception provided by this invention can effectively reduce the interference of poor-performing sensors on the detection results by adding a weight penalty parameter when the area ratio of the detection area is too small during the fusion process of the detection results.

[0041] The hot work monitoring method for construction sites based on deep learning and multi-source image fusion perception provided by this invention can improve the safety level and management efficiency of construction sites and enhance the intelligent management of construction sites by automatically monitoring the working status of hot work operations and improving the accuracy of identification. Attached Figure Description

[0042] Figure 1 This is a flowchart of a construction site hotspot monitoring method based on deep learning and multi-source image fusion perception, according to an embodiment of the present invention. Detailed Implementation

[0043] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, provides a more comprehensive understanding of the construction site hot work monitoring method based on deep learning and multi-source image fusion perception provided by this invention. The advantages and features of this invention will become clearer with the following description. It should be noted that the accompanying drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of this invention.

[0044] like Figure 1As shown in the figure, the construction site hot work monitoring method based on deep learning and multi-source image fusion perception provided in this embodiment includes the following steps:

[0045] S1. Establish target detection networks under visible light and infrared light respectively, and train the target detection networks using hot work operation images to obtain visible light target detection networks and infrared light target detection networks that meet the requirements;

[0046] S2. Use video acquisition equipment to acquire construction site operation videos, and input the construction site operation videos into the visible light target detection network and the infrared light target detection network respectively, identify the distribution of fire and person in each frame of the image, and obtain the bounding box parameters of fire and person respectively;

[0047] S3. Based on the bounding box parameters of the target region obtained in step S2, calculate the difference between the results of the visible light target detection network and the infrared light target detection network, compare them with the threshold, and fuse the detection results;

[0048] S4. Based on the detection results after fusion in step S3, determine whether there is hot work and personnel, and record the relative positions of hot work and personnel.

[0049] This embodiment presents a construction site hot work monitoring method based on deep learning and multi-source image fusion perception. The target detection network, built using deep learning, can identify images captured by construction site monitoring videos, thereby monitoring the hot work status without requiring separate manual monitoring, thus freeing up manpower and demonstrating strong practicality. Furthermore, based on detection materials from visible light and infrared sensors, two target detection networks are used to detect the fire and person areas in the images. By fusing the detection results, the method effectively avoids the inaccuracy of detection from a single image source due to changes in the external environment, improving the accuracy of hot work status identification and reducing the false alarm rate.

[0050] In one specific embodiment, step S1 uses hot work operation images to train the target detection network to obtain a target detection network that meets the requirements, specifically including the following steps:

[0051] S11. Establish a sample database containing images of hot work operations at several construction sites.

[0052] A sufficient number of images of hot work operations are needed to improve training effectiveness.

[0053] S12. Randomly divide the gap images in the sample database into a training set and a test set at a ratio of 9:1. Then, annotate the bounding box parameters and categories of the target areas of the hot work operation images in the training set. The categories include hot work and personnel.

[0054] For the images in the training set, the bounding boxes of the target regions for "fire" and "person" are labeled to form image samples. The target detection network is trained using these image samples until the training parameters meet the requirements.

[0055] S13. Using the established target detection network under visible and infrared light, iteratively train the hot work operation images in the training set, continuously adjusting the network structure and parameters until the trained parameters meet the requirements, thus obtaining a well-trained target detection network.

[0056] The method for training a target detection network provided in this embodiment can obtain a target detection network with high recognition accuracy under visible and infrared light.

[0057] For example, object detection networks may use the YOLOv3 network or other neural networks. The basic network of the YOLOv3 object detection network contains a large number of... and The convolutional layers, named Darknet53 because there are 53 convolutional layers in total, consist of 52 convolutional layers and 1 fully connected layer. The 52 convolutional layers are composed of a 32-pixel array of size 1. The filter consists of a convolution kernel and five sets of repeating residual units. Each of these five sets of residual units contains a single microcontroller of size [size missing]. A convolutional kernel with a stride of 2 is used in a set of repeatedly executed convolutional layers. These layers are executed 1, 2, 8, 8, and 4 times respectively. Within each repeated convolutional layer, the first step is to execute... A convolution operation with a stride of 1 is then performed. For convolution operations with a stride of 1, the number of convolution kernels is first halved and then restored.

[0058] For example, the hot work operation image sample database established in step S1 includes 506 hot work operation images, and the size of the hot work operation images is [missing information]. The units are 640 and 368 pixels. The hot work operation images in the sample database are randomly divided into training and testing sets at a 9:1 ratio. The bounding box parameters and categories of the target regions are then labeled on the hot work operation images in the training set. The categories include "fire" and "person," where the "fire" region is the area where the hot work flame is located, and the "person" region is the area where the hot work personnel are located. The personnel's location is usually around the perimeter of the hot work area. An example of the labeled target region is provided. The network is iteratively trained on the hot work operation images in the training set, continuously adjusting the network structure and parameters until the average loss value is less than 10 and no longer decreases significantly, resulting in a well-trained target detection network.

[0059] In one specific embodiment, the bounding box parameters for the fire and person obtained in step S2 include four-dimensional position parameters. and one-dimensional confidence parameters Among them, These represent the left boundaries of the target region's bounding box. x Coordinates, right boundary x Coordinates, upper boundary y Coordinates, lower boundary y The coordinates are given by the origin of the coordinate system located at the top left corner of the input surveillance image.

[0060] In one specific embodiment, the difference in the result in step S3 is the difference in the bounding box parameters between the visible light target detection network and the infrared light target detection network. This is compared with a threshold, which can be a set difference or the ratio of the difference to the side length. When the difference is within the threshold range, the average value is used for fusion, or the larger value is used for fusion. If the difference exceeds the threshold, a weighting coefficient is set, and fusion is performed.

[0061] In one specific embodiment, the result difference in step S3 refers to the difference in the center point coordinates and area of ​​the bounding box parameters between the visible light target detection network and the infrared target detection network. Calculating the result difference between the visible light target detection network and the infrared target detection network, comparing it with a threshold, and fusing the detection results specifically includes the following steps:

[0062] S31. Calculation of bounding box center point: Based on the bounding box parameters of the detection area under visible light and infrared light obtained in step S2. Calculate the center point coordinates and area of ​​each bounding box using the following formulas:

[0063] ;

[0064] ;

[0065] ;

[0066] ;

[0067] ;

[0068] ;

[0069] in, and These represent the coordinates of the center points of the bounding boxes for open flame and personnel detected by the target detection network under visible and infrared light, respectively. These are the four-dimensional position parameters of the bounding box under visible light. These are the four-dimensional position parameters of the bounding box under infrared light; and These represent the areas of the bounding boxes for open flames and personnel detected by the target detection network under visible and infrared light, respectively.

[0070] S32. Boundary box area ratio calculation: Based on the area of ​​the boundary boxes of hot work and personnel detected by the target detection network under visible and infrared light, calculate the ratio of their area to the weighted sum of areas. The calculation formula is as follows:

[0071] ;

[0072] ;

[0073] in and These represent the proportions of the area of ​​the bounding boxes for open flames and personnel detected by the target detection network under visible and infrared light to the weighted sum of areas, respectively. The threshold representing the area ratio; This is the penalty coefficient.

[0074] S33. Calculate the parameters of the fused bounding box. Based on the area ratio obtained in S32, fuse the target detection results under visible light and infrared light to obtain the predicted bounding boxes for hot work and personnel. The calculation formula is as follows:

[0075] ;

[0076] ;

[0077] ;

[0078] ;

[0079] in and These represent the center coordinates of the bounding box of the fused detection result. and These represent the width and height of the bounding box of the fused detection result, respectively.

[0080] For example, the bounding box parameters of the fire region identified by a visible light target detection network in an input image are {241,356,126,188}, with a confidence level of 0.80; the bounding box parameters of the person region are {318,486,139,174}, with a confidence level of 0.94. The bounding box parameters of the fire region identified by an infrared target detection network are {276,364,69,202}, with a confidence level of 0.85; the bounding box parameters of the person region are... The parameters of the bounding box for the "fire" region in the fused detection results are {314,336,168,227}, with a confidence level of 0.98. The parameters of the bounding box for the "person" region are {256,360,101,194}, with a confidence level of 0.80. The parameters of the bounding box for the "person" region are {316,403,155,203}, with a confidence level of 0.94.

[0081] It should be noted that the fused detection result in step S3 is the fusion of the detection results of the fire region identified by the visible light target detection network and the detection results of the fire region identified by the infrared light target detection network, and the fusion of the detection results of the person region identified by the visible light target detection network and the detection results of the person region identified by the infrared light target detection network. The fused detection result is a new bounding box. The image within the new bounding box can be detected by a target detection network under visible light and / or infrared light to determine whether there is fire and personnel, and record the relative positions of fire and personnel.

[0082] This embodiment provides a method for fusing the detection results of visible light target detection network and infrared light target detection network. This method can eliminate the defects of inaccurate detection caused by monitoring errors due to single light source recognition, improve the accuracy of hot work status recognition, and reduce the false alarm rate. Moreover, by adding a weighted penalty parameter when the detection area ratio is too small, the interference of poorly performing sensors on the detection results can be effectively reduced.

[0083] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0084] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A construction site fire monitoring method based on deep learning and multi-source image fusion perception, characterized in that, Comprise the following steps: S1. respectively establish the target detection network under visible light and infrared light, use the image of fire operation to train the target detection network, obtain the visible light target detection network and infrared light target detection network meeting the requirements; S2. use the video acquisition device to collect the construction site operation video, and input the construction site operation video into the visible light target detection network and infrared light target detection network respectively, identify the distribution of fire and person in each frame of image respectively, and obtain the boundary box parameters of fire and person; S3. according to the boundary box parameters of the target area obtained in step S2, the difference between the results of visible light target detection network and infrared light target detection network is calculated, compared with the threshold value, and the detection results are fused; S4. according to the fusion detection result of step S3, it is judged whether there is fire and personnel, and the relative position of fire and personnel is recorded; Among them, the bounding box parameters of the fire and person obtained in step S2 include four-dimensional position parameters. and one-dimensional confidence parameters ; These represent the left boundaries of the target region's bounding box. x Coordinates, right boundary x Coordinates, upper boundary y Coordinates, lower boundary y Coordinates, where the origin is located at the top left corner of the input surveillance image; In step S3, the difference between the results of visible light target detection network and infrared light target detection network is calculated, compared with the threshold value and fused detection result, which comprises the following steps: S31. Calculate the center point of the bounding box: according to the bounding box parameters of the detection area under visible light and infrared light obtained in step S2 , the center point coordinates and the area of the respective bounding box are calculated respectively, and the calculation formula is as follows: ； ； ； ； ； ； wherein, and respectively represent the center point coordinates of the fire and personnel bounding boxes detected by the target detection network under visible light and infrared light; is a four-dimensional position parameter of the bounding box under visible light, is a four-dimensional position parameter of the bounding box under infrared light; and are the areas of the fire and personnel bounding boxes detected by the target detection network under visible light and infrared light, respectively. S32. boundary box area ratio calculation: according to the area of fire and personnel boundary box detected by the target detection network under visible light and infrared light, the proportion of the weighted area sum is calculated, and the calculation formula is as follows: ； ； wherein and respectively represent the proportion of the area of the fire and personnel bounding box detected by the target detection network under visible light and infrared light to the sum of the weighted areas; represents the threshold of the area ratio; is a penalty coefficient; S33. fusion boundary box detection result calculation, according to the area ratio obtained by S32, the target detection results under visible light and infrared light are fused, and the predicted fire and personnel boundary box is obtained, and the calculation formula is as follows: ； ； ； ； wherein and respectively represent the center coordinates of the fused detection result bounding box, and respectively represent the width and height of the fused detection result bounding box.

2. The deep learning and multi-source image fusion perception-based construction site fire monitoring method of claim 1, wherein, Step S1 uses the image of fire operation to train the target detection network, and obtains the target detection network meeting the requirements, which comprises the following steps: S11. establish a sample database containing a plurality of construction site fire operation images; S12. the gap image in the sample database is randomly divided into training set and test set according to the predetermined proportion, and the boundary box parameters and category of target area in the fire operation image in the training set are labeled, the category includes fire and personnel; S13. use the established target detection network under visible light and infrared light to iteratively train the fire operation image in the training set, and constantly adjust the structure and parameters of the network until the training parameters meet the requirements, and obtain the trained target detection network.

3. The deep learning and multi-source image fusion perception-based construction site fire monitoring method of claim 1, wherein, The target detection network adopts YOLOv3 network.

Images