Fire safety risk early warning method and system based on visual perception

Abstract

The present invention proposes a fire safety risk warning method and system based on visual perception. The method includes: inputting a fire safety risk index sequence and a ROI area image of a hot work operation into a first network branch, outputting a first hot work operation detection scene complexity and a guidance parameter sequence; inputting a wide-angle image of the hot work operation area into a second network branch, obtaining a first regression value sequence and a second regression value sequence; inputting a wide-angle image of the hot work operation area into a third network branch, the third network branch performs convolution according to the guidance parameter sequence, the first regression value sequence, and the second regression value sequence, and outputs a second hot work operation detection scene complexity and a fire safety risk index abnormal value; and generating fire safety warning information according to the fire risk index abnormal value. Adaptively adjust the number of longitudinal convolutions and transverse convolutions when extracting the hot work operation image features, remove redundant information and improve detection accuracy, obtain more accurate fire risk index abnormal values, and generate warning information.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and production safety, and specifically to a fire safety risk method and system based on visual perception. Background Art

[0002] Fire safety is of vital importance in enterprise safety. Hot work during daily maintenance of enterprises is prone to cause safety accidents and poses potential safety risks. Hot work usually refers to welding and cutting operations in fire-free areas defined by chemical enterprises and temporary operations using blowtorches, electric drills, grinding wheels, etc. in flammable and explosive places that may produce flames, sparks and hot surfaces. The requirements for hot work operations will be different in different environments. Conversely, hot work operations have different requirements for the surrounding environment. The operating actions of the operators also have a certain impact on the safety of hot work operations. At present, the main detection method mainly relies on manual monitoring to detect the operation process.

[0003] The problem with existing detection methods is that manual detection may be highly subjective, making it difficult to accurately determine whether there is a fire safety risk in the hot work at this moment. In addition, manual detection has a certain lag and cannot achieve timely perception, resulting in untimely warning of fire safety risks, which brings safety hazards. Summary of the invention

[0004] In order to solve the above technical problems, the purpose of the present invention is to provide a fire safety risk early warning method and system based on visual perception. The technical solutions adopted are as follows:

[0005] In a first aspect, an embodiment of the present invention provides a fire safety risk early warning method based on visual perception.

[0006] Input the fire safety risk index sequence and the hot work ROI area image into the first network branch, and output the first hot work detection scene complexity and guidance parameter sequence;

[0007] Input the wide-angle image of the hot work area into the second network branch to obtain a first regression value sequence and a second regression value sequence;

[0008] The wide-angle image of the hot work area is input into the third network branch. The third network branch performs convolution according to the guidance parameter sequence, the first regression value sequence, and the second regression value sequence, and outputs the complexity of the second hot work detection scene and the abnormal value of the fire safety risk index; and generates fire safety warning information based on the abnormal value of the fire risk index.

[0009] Preferably, the structure of the first network branch includes: extracting features of the fire safety risk indicator sequence through the first fully connected layer to obtain a first feature tensor; sending the hot work ROI area image to the first encoder to extract features and obtain a second feature tensor; concatenating the first feature tensor and the second feature tensor to obtain a third feature tensor; sending the third feature tensor to the second encoder to obtain a fourth feature tensor; the fourth feature tensor is respectively sent to the second fully connected layer and the first decoder; the second fully connected layer outputs the complexity of the hot work detection scene, and the first decoder outputs the guidance parameter sequence.

[0010] Preferably, the second network branch structure includes: sending the wide-angle image of the hot work area to the third encoder to obtain the fifth feature tensor; the fifth feature tensor outputs the first regression value sequence and the second regression value sequence through the third fully connected layer.

[0011] Preferably, the structure of the third network branch includes: sending the wide-angle image of the hot work area to the fourth encoder to obtain the sixth feature tensor; sending the sixth feature tensor to the fourth fully connected layer and the fifth fully connected layer respectively, the fourth fully connected layer outputs the complexity of the second hot work detection scene, and the fifth fully connected layer outputs the abnormal value of the fire safety risk indicator.

[0012] Preferably, each element in the guidance parameter sequence represents a convolution kernel, including two dimensions: a horizontal convolution size and a vertical convolution size; each element in the guidance parameter sequence corresponds to a first regression value and a second regression value, respectively, and the first and second regression values ​​are used to control the number of convolutions of the convolution kernel.

[0013] In a second aspect, an embodiment of the present invention provides a fire safety risk warning system based on visual perception.

[0014] A fire safety risk early warning system based on visual perception, characterized in that the system comprises: a module for acquiring the complexity of a hot work detection scene and a guidance parameter sequence, which is used to input a fire safety risk indicator sequence and a hot work ROI area image into a first network branch, and output a first hot work detection scene complexity and a guidance parameter sequence;

[0015] A regression value sequence acquisition module is used to input the wide-angle image of the hot work area into the second network branch to obtain a first regression value sequence and a second regression value sequence;

[0016] The fire safety risk index acquisition module is used to input the wide-angle image of the hot work area into the third network branch. The third network branch performs convolution according to the guidance parameter sequence, the first regression value sequence, and the second regression value sequence, and outputs the complexity of the second hot work detection scene and the fire safety risk index abnormal value; and generates fire safety warning information according to the fire risk index abnormal value.

[0017] The technical solution of the present invention has the following beneficial effects:

[0018] The present invention not only improves the training accuracy of the network through the auxiliary branches in the fire safety risk warning network composed of the first, second and third network branches, but also ensures that the effect of the asymmetric convolution kernel of the third network branch in extracting features is not weaker than the symmetric convolution kernel; according to the actual type of hot work, the number of longitudinal spatial convolution and transverse spatial convolution of the multi-scale asymmetric convolution kernel when extracting image features is determined, which can effectively remove redundant information and improve detection accuracy. Moreover, the features extracted by the multi-scale asymmetric convolution kernel will be richer, so that the obtained fire safety risk warning information will be more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0020] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of a fire safety risk warning method and system based on visual perception proposed by the present invention, its specific implementation method, structure, features and effects, in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures, or characteristics in one or more embodiments may be combined in any suitable form.

[0021] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0022] Embodiment 1:

[0023] The specific scenario targeted by the present invention is: in the hot work scenario during the production and maintenance process of the enterprise, since hot work will bring about high temperature impact, and the fire safety risk index of hot work changes dynamically according to the actual operation, the hot work involves environmental monitoring and operator operation monitoring; a gun-ball linked camera group is deployed in the hot work scenario, and the camera types include IR camera and RGB camera.

[0024] The following is a detailed description of a fire safety risk early warning method based on visual perception provided by the present invention in conjunction with the accompanying drawings.

[0025] Firstly, the fire safety risk index sequence and the hot work ROI area image are input into the first network branch, and the complexity of the first hot work detection scene and the guidance parameter sequence are output.

[0026] In this step, the IR camera is used for preliminary perception, and the ball camera is called according to the perception result, which improves the timeliness of detection and can obtain accurate and high-resolution images of the hot work area for subsequent processing. Specifically, the gun camera is used to obtain thermal imaging images. The gun camera includes an IR camera and an RGB camera to collect wide-angle infrared thermal imaging images and RGB images. For infrared thermal imaging images, a temperature threshold m is set. T , according to m T Perform threshold segmentation and then connective domain analysis to obtain several suspicious high-temperature areas.

[0027] The type of hot work is judged, specifically: based on the information of multiple consecutive frames of thermal imaging images in time sequence, the corresponding category of the suspected high-temperature area is judged, which includes three categories, namely flame, spark and hot surface; when the suspected high-temperature area appears continuously and the area changes, it is judged to be the flame category; when the suspected high-temperature area appears intermittently and discretely, it is judged to be the spark category; when the suspected high-temperature area appears continuously and the area is stable, it is judged to be the hot surface category; and the gun-ball linkage technology is used to mobilize the ball camera to collect the RGB image of the corresponding position. It should be noted that the gun-ball linkage technology is an existing technology, which can obtain a target point in a wide-angle view. The target point is the centroid point of the connected domain, and the ball camera is controlled to collect the RGB image corresponding to the target point.

[0028] The operations of hot work are identified as follows: the human skeleton image is obtained through the open source project OpenPose, the semantic segmentation image is obtained through the semantic segmentation network, and multiple frames of continuous human skeleton images and semantic segmentation images are sent as input to TCN to extract features. The feature tensor is expanded and sent to the fully connected layer, and the operation classification result is output through the SoftMax function. When the network is trained, the training set is multiple groups of continuous multiple frames of hot work images of different categories. After processing, the human skeleton images and semantic segmentation images obtained are labeled into five categories, including welding operations (welding, cutting), open flame operations (using blowtorches, stoves, liquefied gas stoves, electric stoves, etc.), construction operations (steeling pipes, boiling asphalt, frying sand, etc.), physical operations (grinding, sandblasting, hammering, etc. in flammable and explosive areas), and others; the labels need to be one-hot encoded, and the loss function adopts the cross entropy loss function.

[0029] The generation of the ROI area for hot work is specifically as follows: the positions of the gun camera and the ball camera are known, and the homography matrix corresponding to their transformation is known. The key points extracted based on the ball camera RGB image are projected into the gun camera wide-angle RGB image, and the key point pixel values ​​are assigned according to their hot work categories. The five categories of hot work categories are numbered 1, 2, 3, 4, and 5. For example, the key point pixel value of the welding operator is consistent with the hot work category number; at this time, in order to distinguish the key points from the actual RGB image pixels, the pixel values ​​of the pixels around the key points are changed to conform to the Gaussian distribution; the wide-angle RGB image after the above processing is used as input and sent to the regression network to regress the hot work ROI area. The hot work ROI area is a bounding box, and the coordinates of the upper left corner and the lower right corner of the bounding box are specifically regressed; it should be noted that due to the complexity of hot work monitoring, the annotation information of the network needs to be annotated by experienced staff to ensure the accuracy of the hot work ROI area.

[0030] The hot work ROI area image is obtained as follows: the bounding box obtained by regression is used to intercept the hot work ROI area image from the wide-angle image of the hot work area.

[0031] The fire safety risk index is obtained by: realizing it through a mapping table, that is, based on the provisions of hot work in production safety, the corresponding fire safety risk index is determined according to the hot work type and personnel operation information, and stored in the mapping table. For example, when the hot work type is sparks and the personnel operation information is open flame operation, the fire safety risk index includes air circulation, temperature, flammable gas concentration, etc. In actual use, the actual corresponding fire safety risk index is obtained by searching the mapping table according to the actual hot work type and personnel operation information.

[0032] The fire safety warning of this embodiment is implemented through the fire safety risk warning network. The first network branch of the fire safety risk warning network is specifically constructed as follows:

[0033] The fire safety risk index sequence is extracted with features through the first fully connected layer to obtain the first feature tensor; the image of the ROI area of ​​the hot work is sent to the first encoder to extract features and obtain the second feature tensor. The first feature tensor is consistent with the second feature tensor in size. After the Concat operation, the third feature tensor is obtained. The third feature tensor is sent to the second encoder to obtain the fourth feature tensor. The fourth feature tensor is sent to the second fully connected layer and the first decoder respectively. The second fully connected layer outputs the first detection scene complexity through the Softmax function; the first decoder outputs a guidance parameter sequence, which is consistent with the length of the first regression value sequence and the second regression value sequence, and contains two dimensions. The guidance parameter is used to determine the convolution kernel size of the second network branch. In particular, in this embodiment, the guidance parameter sequence has the same length as the fire safety risk index sequence.

[0034] Next, the wide-angle image of the hot work area is input into the second network branch to obtain the first regression value sequence and the second regression value sequence.

[0035] The second network branch of the fire safety risk early warning network is constructed as follows:

[0036] The wide-angle image of the hot work area after the pixel value change is sent to the third encoder, and the third encoder outputs the fifth feature tensor. The fifth feature tensor outputs the first regression value sequence through the third fully connected layer. and the second regression value series

[0037] Finally, the wide-angle image of the hot work area is input into the third network branch. The third network branch performs convolution according to the guidance parameter sequence, the first regression value sequence, and the second regression value sequence, and outputs the complexity of the second hot work detection scene and the abnormal value of the fire safety risk index; fire safety warning information is generated according to the abnormal value of the fire risk indicator.

[0038] The third network branch of the fire safety risk early warning network is constructed as follows:

[0039] The wide-angle image of the hot work area is sent to the fourth encoder to obtain the sixth feature tensor; the sixth feature tensor is sent to the fourth fully connected layer and the fifth fully connected layer respectively, the fourth fully connected layer outputs the complexity of the second hot work detection scene, and the fifth fully connected layer outputs the abnormal value of the fire risk index. Then, fire safety warning information is generated according to the abnormal value of the fire risk index.

[0040] It should be noted that the convolution layer operation in the fourth encoder in the third network branch adopts a multi-scale asymmetric convolution kernel, that is, the convolution kernel size is M t *N t , where t represents the tth guiding parameter in the guiding parameter sequence; the multi-scale asymmetric convolution kernel is split into a 1*N one-dimensional horizontal convolution kernel and an M*1 one-dimensional vertical convolution kernel; each element in the guiding parameter sequence corresponds to a first regression value and a second regression value, and the first and second regression values ​​are used to control the number of convolutions of the convolution kernel. First regression value sequence Guide M t *1 horizontal convolution kernel performs convolution horizontally and controls the number of convolutions. The second regression value sequence Guide 1*N tThe longitudinal convolution kernel is convolved longitudinally and the number of convolutions is controlled; then the features obtained by the transverse convolution and the longitudinal convolution are fused through the Concat operation. Multi-scale convolution is an asymmetric convolution, that is, the convolution kernel of size M*N is convolved according to the one-dimensional convolution kernels of M*1 and 1*N. Accordingly, for the same pixel point, the conventional single convolution is decomposed into at least two convolutions, that is, the sum of the number of convolutions K1 of the longitudinal spatial convolution based on the M*1 convolution kernel and the number of convolutions K2 of the transverse spatial convolution based on the 1*N convolution kernel. The purpose of the decomposition here is to distinguish the representation dimensions of features in different directions of the spatial domain. Since actual hot work operations require higher detection accuracy, the above method can effectively improve the accuracy of detection. In the present invention, a convolution refers to a complete convolution of the entire feature map. If the convolution layer is not iterated, the number of convolutions is the number of convolution layers.

[0041] In the fire safety risk warning network structure, the second fully connected layer and the fourth fully connected layer are auxiliary branches used to provide additional supervision during training. The specific method of network training is as follows: the training set uses multiple groups of data under different hot work categories, and the label data are the manually labeled hot work detection scene complexity and the manually labeled fire safety risk indicator anomalies.

[0042] The loss function of the first network branch includes the second fully connected layer loss L1, which uses the cross entropy loss function; the loss function of the second network branch includes the regression loss L2 of the third fully connected layer, which needs to combine the bounding box information W, H and the reference information M t ,N t Construct and obtain the ratio of the image size of the ROI area of ​​the hot work to the dimension of the corresponding element of the guidance parameter sequence, calculate the mean square error between the ratio and the corresponding regression value, use the spatial perception category to correct the mean square error, and sum all the corrected values ​​to get the loss of the second network branch. Specifically:

[0043]

[0044] Among them, [·] is a rounding operation, β is a scaling value, which is set larger and set to 100, c is the spatial perception category, and the category values ​​are -1, 0, 1, which is used to control the weight relationship of the convolution spatial direction. It should be noted that the spatial perception category is obtained by mapping the hot work category to the spatial perception category. The mapping relationship is set by professional staff based on work experience. While ensuring the receptive field, spatial direction features at different levels are extracted to improve detection accuracy; the loss function of the third network branch includes the regression loss L3 of the fifth fully connected layer, and the mean square error loss function is adopted; the second fully connected layer and the fourth fully connected layer are auxiliary branches, and their contrast loss function is out4 represents the output value of the fourth fully connected layer, and out2 represents the output category representation of the second fully connected layer.

[0045] In the fire safety risk warning network structure, the advantage of the multi-scale asymmetric convolution kernel is that it extracts features based on actual working conditions, removes redundant information, improves detection accuracy, and extracts features according to different scales, making the features richer; the advantage of the auxiliary branch is that it ensures that the effect of feature extraction by the asymmetric convolution kernel is not weaker than that of the symmetric convolution kernel.

[0046] Example 2

[0047] This embodiment provides a system embodiment. A fire safety risk warning system based on visual perception, comprising: a hot work detection scene complexity and guidance parameter sequence acquisition module, which is used to input the fire safety risk indicator sequence and the hot work ROI area image into a first network branch, and output the first hot work detection scene complexity and guidance parameter sequence;

[0048] A regression value sequence acquisition module is used to input the wide-angle image of the hot work area into the second network branch to obtain a first regression value sequence and a second regression value sequence;

[0049] The fire safety risk index acquisition module is used to input the wide-angle image of the hot work area into the third network branch. The third network branch performs convolution according to the guidance parameter sequence, the first regression value sequence, and the second regression value sequence, and outputs the complexity of the second hot work detection scene and the abnormal value of the fire safety risk index; and generates fire safety warning information according to the abnormal value of the fire risk index.

[0050] The above description is only a preferred embodiment of the present invention and is not intended to limit the spirit and principle of the present invention. Any modifications, equivalent substitutions, improvements, etc. made should be included in the protection scope of the present invention.

Claims

1. A fire safety risk early warning method based on visual perception, characterized in that: The method includes: Input the fire safety risk index sequence and the hot work ROI area image into the first network branch, and output the first hot work detection scene complexity and guidance parameter sequence; Input the wide-angle image of the hot work area into the second network branch to obtain a first regression value sequence and a second regression value sequence; The wide-angle image of the hot work area is input into the third network branch, and the third network branch performs convolution according to the guidance parameter sequence, the first regression value sequence, and the second regression value sequence, and outputs the complexity of the second hot work detection scene and the abnormal value of the fire safety risk index; Generate fire safety warning information based on abnormal values ​​of fire risk indicators; The wide-view image of the hot work area is sent to the third encoder to obtain the fifth feature tensor; the fifth feature tensor outputs the first regression value sequence and the second regression value sequence through the third fully connected layer; the wide-view image of the hot work area includes a wide-view infrared thermal imaging image and an RGB image; Each element in the guidance parameter sequence represents a convolution kernel, including two dimensions: horizontal convolution size and vertical convolution size; each element in the guidance parameter sequence corresponds to a first regression value and a second regression value, and the first and second regression values ​​are used to control the number of convolutions of the convolution kernel; the convolution layer operation in the fourth encoder in the third network branch adopts a multi-scale asymmetric convolution kernel, that is, the convolution kernel size is ,in Represents the first guidance parameters; split the multi-scale asymmetric convolution kernel into The one-dimensional horizontal convolution kernel and One-dimensional vertical convolution kernel; each element in the guidance parameter sequence corresponds to a first regression value and a second regression value, and the first and second regression values ​​are used to control the number of convolutions of the convolution kernel; the first regression value sequence guide The horizontal convolution kernel is convolved horizontally and the number of convolutions is controlled. The second regression value sequence guide The longitudinal convolution kernel is convolved longitudinally and the number of convolutions is controlled; the features obtained by the horizontal and longitudinal convolutions are then fused through the Concat operation.

2. The method according to claim 1, characterized in that The structure of the first network branch includes: extracting features of the fire safety risk indicator sequence through the first fully connected layer to obtain a first feature tensor; sending the hot work ROI area image to the first encoder to extract features and obtain a second feature tensor; concatenating the first feature tensor and the second feature tensor to obtain a third feature tensor; sending the third feature tensor to the second encoder to obtain a fourth feature tensor; the fourth feature tensor is respectively sent to the second fully connected layer and the first decoder; the second fully connected layer outputs the complexity of the first hot work detection scene, and the first decoder outputs a guidance parameter sequence.

3. The method according to claim 1, characterized in that The structure of the third network branch includes: sending the wide-angle image of the hot work area to the fourth encoder to obtain the sixth feature tensor; sending the sixth feature tensor to the fourth fully connected layer and the fifth fully connected layer respectively, the fourth fully connected layer outputs the complexity of the second hot work detection scene, and the fifth fully connected layer outputs the abnormal value of the fire safety risk indicator.

4. A fire safety risk early warning system based on visual perception, characterized in that: The system includes: a hot work detection scene complexity and guidance parameter sequence acquisition module, which is used to input the fire safety risk indicator sequence and the hot work ROI area image into the first network branch, and output the first hot work detection scene complexity and guidance parameter sequence; A regression value sequence acquisition module is used to input the wide-angle image of the hot work area into the second network branch to obtain a first regression value sequence and a second regression value sequence; The fire safety risk index acquisition module is used to input the wide-angle image of the hot work area into the third network branch, and the third network branch performs convolution according to the guidance parameter sequence, the first regression value sequence, and the second regression value sequence, and outputs the complexity of the second hot work detection scene and the abnormal value of the fire safety risk index; and generates fire safety warning information according to the abnormal value of the fire risk index; The regression value sequence acquisition module is also used to send the wide-angle image of the hot work area to the third encoder to obtain the fifth feature tensor; the fifth feature tensor outputs the first regression value sequence and the second regression value sequence through the third fully connected layer; the wide-angle image of the hot work area includes a wide-angle infrared thermal imaging image and an RGB image; Each element in the guidance parameter sequence represents a convolution kernel, including two dimensions: horizontal convolution size and vertical convolution size; each element in the guidance parameter sequence corresponds to a first regression value and a second regression value, and the first and second regression values ​​are used to control the number of convolutions of the convolution kernel; the convolution layer operation in the fourth encoder in the third network branch adopts a multi-scale asymmetric convolution kernel, that is, the convolution kernel size is ,in Represents the first guidance parameters; split the multi-scale asymmetric convolution kernel into The one-dimensional horizontal convolution kernel and One-dimensional vertical convolution kernel; each element in the guidance parameter sequence corresponds to a first regression value and a second regression value, and the first and second regression values ​​are used to control the number of convolutions of the convolution kernel; the first regression value sequence guide The horizontal convolution kernel is convolved horizontally and the number of convolutions is controlled. The second regression value sequence guide The longitudinal convolution kernel is convolved longitudinally and the number of convolutions is controlled; the features obtained by the horizontal and longitudinal convolutions are then fused through the Concat operation.

5. The system according to claim 4, characterized in that The module for acquiring the complexity of the hot work detection scene and the guidance parameter sequence is also used to extract features from the fire safety risk indicator sequence through the first fully connected layer to obtain a first feature tensor; send the hot work ROI area image to the first encoder to extract features and obtain a second feature tensor; concatenate the first feature tensor and the second feature tensor to obtain a third feature tensor; send the third feature tensor to the second encoder to obtain a fourth feature tensor; The fourth feature tensor is sent to the second fully connected layer and the first decoder respectively; the second fully connected layer outputs the complexity of the hot work detection scene, and the first decoder outputs the guidance parameter sequence.

6. The system according to claim 4, characterized in that The fire safety risk index acquisition module is also used to send the wide-angle image of the hot work area into the fourth encoder to obtain the sixth feature tensor; the sixth feature tensor is sent to the fourth fully connected layer and the fifth fully connected layer respectively, the fourth fully connected layer outputs the complexity of the second hot work detection scene, and the fifth fully connected layer outputs the abnormal value of the fire safety risk index.

Images