A deep learning-based hazard and violation behavior recognition method

Abstract

A dangerous source and violation behavior recognition method based on deep learning belongs to the technical field of image recognition and safety monitoring. In view of the particularity of flame and violation behavior detection, an improved FSE attention mechanism is proposed. First, an image dataset containing flames and various laboratory violation behaviors is constructed. Then, the dataset is used to train the target detection network, and the improved FSE attention mechanism is introduced into the network to improve the feature extraction capability. Finally, the trained model is deployed to the real-time detection system. The method can analyze in real time through the camera or video file, automatically identify flames and violation behaviors such as not wearing protective articles and playing mobile phones, and timely issue warnings. The method solves the problems of single detection and poor real-time performance of traditional methods, and improves the safety monitoring capability of laboratory and other scenes.

Description

Technical Field

[0001] This invention relates to a method for identifying hazards and violations based on deep learning, which belongs to the field of image recognition and security monitoring technology. Background Technology

[0002] In university laboratories and factories, personnel violations are a major cause of safety accidents. Fire is one of the most common accidents. Currently, safety management mainly relies on manual supervision or traditional sensors. Traditional fire detectors have limitations. For example, smoke detectors are ineffective in open spaces. Heat detectors are greatly affected by airflow. These methods have poor real-time performance and limited coverage.

[0003] Traditional methods for detecting human behavior rely on manual feature extraction. These methods are computationally complex and ill-suited for complex environments. Image recognition-based detection methods are gaining attention. However, existing methods have shortcomings. Many studies only detect flames or only detect behavior. Existing publicly available datasets have limited data on laboratory scenarios. This results in trained models with poor adaptability.

[0004] Therefore, a method is needed that can simultaneously detect hazards and violations. This method should be highly accurate, real-time, and adaptable to complex environments. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a method for identifying hazards and violations based on deep learning, which can simultaneously detect hazards and violations. This method is highly accurate and has a fast response.

[0006] The technical solution adopted in this invention is as follows: A method for identifying hazards and violations based on deep learning, comprising the following steps:

[0007] S1. Dataset Construction: Randomly sample frames to collect images containing flames and violations by laboratory personnel. Label the flames and defined violations in the images. Combine measurement sequence i with image I to form a 3D vector, which constitutes the dataset. Use RGB three-channel format as the input to the neural network. Divide the dataset proportionally into a training set. and verification set ;

[0008] S2. Constructing the detection model: Constructing a target detection neural network, incorporating an improved FSE attention mechanism into the network; selecting ReLU and Sigmoid activation functions; the neural network output is a set containing multiple detection boxes; specifically including the following sub-steps:

[0009] S2.1 When constructing the original model, the input image I and the convolution kernel h are used to output the feature map F. The operation is performed using the following formula:

[0010]

[0011] in, , The coordinates inside the convolution kernel, with values ​​ranging from 0 ≤ m < 0. , 0 ≤ n < , This is the index of the input channel. , Let k be the position coordinates of the input image I, and k be the output channel number. This represents the total number of input channels. , These are the height and width of the convolution kernel, respectively;

[0012] S2.2, the FSE module performs the extrusion operation, inputting the feature map F. The global average pooling result for the c-th channel is:

[0013]

[0014] in, Let H be the compressed feature of the c-th channel, where H and W are the height and width of the feature map, and C is the total channel size. , The coordinates on the c-channel feature map are, i.e. c-channel input feature map The pixel value in the middle coordinate is , point;

[0015] S2.3, Activation Operation: Through two fully connected layers, channel weights are learned.

[0016]

[0017] z is the compressed vector. W1 represents the channel weight vector output after the activation operation; W2 represents the weights of the fully connected layer. It is the ReLU activation function. Use the Sigmoid activation function;

[0018] S2.4, Recalibration operation: Multiply the learned weights by the feature map.

[0019]

[0020] in, This is the enhanced feature map of the c-th channel. Let c be the weight of the c-th channel. Input the feature map for channel c;

[0021] S3. Model Training and Optimization: Using the training set obtained in step S1 Train the constructed neural network model; use the training set Input the data into the network in batches and minimize the loss function. for:

[0022]

[0023] in, The weights are the bounding box loss. Weight the target confidence level loss. For classification loss weights;

[0024] The following CIoU formula is used to optimize the proximity of the output bounding box to the actual bounding box in terms of position and size:

[0025]

[0026] In the formula, The distance is represented by the center point, and l is the length of the diagonal of the minimum bounding box. For dynamic weighting coefficients, ,in ;

[0027] W and h are the width and height of the predicted bounding box. , These are the actual frame width and height, respectively;

[0028] Intersection over Union (IoU) represents the degree of overlap between the predicted bounding box and the manually annotated region, and is expressed by the following formula:

[0029]

[0030] B and These are the predicted bounding box and the ground truth bounding box, respectively.

[0031] To optimize the model's ability to determine whether a target exists, a binary cross-entropy loss function is used:

[0032]

[0033] N is the total number of predicted locations; For the first The actual situation at each location, with a value of 0 for no target and a value of 1 for a target; For the model to the first The probability of predicting a location;

[0034] To optimize the model's ability to identify target categories, a multi-class cross-entropy loss function is used:

[0035]

[0036] N is the total target number; Let q be the category of the i-th target, where the value of the true category q is 1 and the value of the non-true category is 0; The probability predicted by the model is O, where O is the total number of categories and o represents the o-th category.

[0037] During model training, this invention introduces an early stopping strategy to improve the model's generalization ability and prevent overfitting. Figure 3 This demonstrates the complete process of this training and early stopping strategy: First, the model parameters are initialized and the training set data is loaded. Then, in each training round, forward propagation, loss calculation, and backpropagation are performed sequentially to update the weights, and the training set loss is calculated. Next, the model is evaluated using an independent validation set, and the validation loss is calculated. An early stopping mechanism is used (if the validation loss does not reach a new low for 20 consecutive rounds, training is terminated, and the historical best weights are loaded) to prevent overfitting. Finally, the model performance is evaluated on the test set, and the trained model is output. The entire process takes into account both model optimization and generalization ability.

[0038] The specific implementation method is as follows: after each training round, calculate the model on the validation set. The model calculates the lowest validation loss value and records the historical minimum validation loss. If the validation loss does not fall below the historical minimum value for 20 consecutive rounds, the model is considered to have converged, and training is terminated early. After training, the model weights at the point of lowest validation loss are saved as the final model.

[0039] S4. Deploy the trained model into the real-time detection system to identify and warn of dangerous sources and violations in the input video stream in real time.

[0040] Furthermore, in step S1, the resolution and format of the image are consistent, and the pixel values ​​are normalized to the range [0,1].

[0041] Furthermore, in step S1, the random frame sampling specifically involves extracting image frames from the video at fixed intervals of N frames, where N is 30.

[0042] Furthermore, in step S1, the personnel violations include not wearing protective equipment, eating, drinking, playing on mobile phones, and sleeping.

[0043] Furthermore, in step S2, each detection box has four parameters: bounding box position, width and height, confidence level, and category.

[0044] Furthermore, in step S2, the improved FSE attention mechanism module includes: (1) a multi-scale feature perception unit, which uses multi-branch fusion of global pooling, local pooling and edge pooling; (2) a color perception channel grouping unit, which processes feature channels according to color sensitivity; (3) a spatiotemporal joint attention unit, which processes the temporal dynamic features of continuous frame sequences; and (4) a small target spatial recalibration unit, which weights the small target region through a spatial importance map.

[0045] Alternatively, it may include edge-enhancing guidance units to improve the recognition of transparent or reflective objects.

[0046] Furthermore, the output of the multi-scale feature sensing unit is calculated as follows:

[0047]

[0048] In the formula: , , Defined as multi-scale pooling branch weights, and =1, For global average pooling, the following formula is used:

[0049]

[0050] For local pooling, the following formula is used:

[0051]

[0052] For edge feature pooling, the following formula is used:

[0053] .

[0054] Furthermore, the small target spatial recalibration unit specifically includes constructing a spatial importance graph:

[0055]

[0056] Weighted enhancement is applied to small target areas:

[0057]

[0058] For position ( , The spatial importance value at position () reflects the salience of that position in the feature map, where C is the total number of channels in the feature map. This is the enhanced feature map of the c-th channel.

[0059] Furthermore, the edge enhancement guidance unit specifically includes extracting edge feature maps;

[0060]

[0061] Sobel refers to the Sobel operator;

[0062] Incorporate edge features into channel weights:

[0063]

[0064] Defined as the edge feature map of the c channel. The weight of the c-th channel after edge-guided enhancement. This represents the weight of the c-th channel in the original attention mechanism; This is the edge enhancement coefficient, used to control the degree of influence of edge information on the weights.

[0065] The beneficial effects of this invention are as follows:

[0066] This method and system are primarily applied in laboratories, factories, and other locations requiring security monitoring. The improved FSE attention module enhances the model's ability to distinguish complex dynamic flames and simultaneously improves its ability to identify small, reflective, and transparent prohibited items. Specifically, this is reflected in:

[0067] 1. By fusing global, local, and edge pooling features, the stability of identifying multi-scale and multi-morphological targets such as flames is significantly improved;

[0068] 2. To address the gradual change characteristics of flame color, the weights of color-sensitive channels were learned in groups, enhancing the adaptability to changes in flame color.

[0069] 3. By fusing information from consecutive frames, the dynamic characteristics of flame changes can be effectively captured;

[0070] 4. By constructing a spatial importance map and weighting small target regions, the model's detection accuracy for small-sized illegal items such as mobile phones, water cups, and gloves was significantly improved;

[0071] 5. By using edge features to guide the allocation of attention weights, the model's ability to recognize transparent or reflective objects (such as goggles and transparent bottles) is enhanced, and it performs excellently under complex lighting conditions in the laboratory. Attached Figure Description

[0072] Figure 1 This is a schematic diagram illustrating the working principle of the SE attention mechanism.

[0073] Figure 2 This is a schematic diagram of the SE attention mechanism module structure.

[0074] Figure 3 This is a training process monitoring and early stop strategy diagram.

[0075] Figure 4 It is the result of image processing.

[0076] Figure 5 Schematic diagram of the workflow of the small target space recalibration unit.

[0077] Figure 6 A schematic diagram of the overall structure of the improved FSE attention module. Detailed Implementation

[0078] The specific embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0079] A deep learning-based method for identifying hazards and violations includes the following steps:

[0080] S1. Constructing the dataset: Randomly sample frames to collect images with uniform resolution and format containing flames and violations by laboratory personnel. Label the flames and defined violations in the images. Combine the measurement sequence i with the image signal F to form a 3D vector, which constitutes the dataset. Divide the dataset proportionally into a training set. and verification set .

[0081] S2. Constructing the Detection Model: Constructing an object detection network, incorporating an improved FSE attention mechanism into the network. The improved FSE modules are inserted in a differentiated manner based on the characteristics of different layers of the network.

[0082] S3. Model Training and Optimization: Using the training set obtained in step S1 Train the constructed neural network model.

[0083] S4. System Deployment and Application: Deploy the trained model into the real-time detection system to identify and warn of dangerous sources and violations in the input video stream in real time.

[0084] Furthermore, in step S1, the original image is obtained through two methods: downloading from a website and extracting frames from a network video.

[0085] Furthermore, in step S1, the resolution and format of the image are consistent, and the pixel values ​​are normalized to the range [0,1].

[0086] Furthermore, in step S1, the random frame sampling specifically involves extracting image frames from the video at fixed intervals of N frames, where N is 30 frames.

[0087] Furthermore, in step S1, the personnel violations include not wearing protective equipment, eating, drinking, playing on mobile phones, and sleeping.

[0088] Further, in step S2, the targets in the image are labeled with rectangular boxes using annotation software. The annotation categories include "flame," "gloves," "goggles," "laboratory clothing," "mask," "tableware," "water cup," and "mobile phone." Further, in step S2, the model in question is the base model.

[0089] Furthermore, in step S2, each detection box has four parameters: bounding box position, width and height, confidence level, and category.

[0090] Furthermore, in step S4, the warning method includes drawing a marker box in the video frame and issuing an audio prompt.

[0091] A deep learning-based system for identifying hazards and violations, used to implement the above method, includes:

[0092] The data acquisition and processing module is used to acquire and label image data and construct a dataset. The model training module loads the network, incorporates an improved FSE attention mechanism, and trains it using the dataset. The real-time detection module loads the trained model and performs real-time analysis and recognition of the input video stream. The human-computer interaction module provides a graphical interface for parameter setting, result display, and alarm management.

[0093] Example 1

[0094] A method for identifying hazards and violations based on deep learning, specifically including the following steps:

[0095] S1. Dataset Construction: Randomly sampled frames to collect images containing flames and violations by laboratory personnel. The flames and defined violations in the images are labeled. The measurement sequence i is combined with the image signal F to form a 3D vector, constituting the dataset. RGB three-channel format is used as the input to the neural network. The dataset is proportionally divided into a training set. and verification set ;

[0096] S2. Constructing the detection model: Constructing a target detection neural network, incorporating an improved FSE attention mechanism into the network; the FSE module is optimized in multiple dimensions to suit the specific characteristics of the detected target, using ReLU and Sigmoid activation functions; the neural network output is a set containing multiple detection boxes; specifically including the following sub-steps:

[0097] S2.1 When constructing the original model, the input image I and the convolution kernel h are used to output the feature map F. The operation is performed using the following formula:

[0098]

[0099] in, The coordinates are inside the convolution kernel. This is the input channel number. , Here, k represents the position coordinates of the input image, and k represents the output channel number. This represents the total number of input channels. , These represent the height and width of the convolution kernel, respectively.

[0100] S2.2 Improve the FSE module. In the original extrusion operation, the input feature map F The global average pooling result for the c-th channel is:

[0101]

[0102] in, Let H be the compressed feature of the c-th channel, and H and W be the height and width of the feature map. , Input the coordinates on the image for channel c.

[0103] S2.2.1 Multi-scale feature sensing unit: A multi-branch pooling strategy is adopted to address the various characteristics of flames.

[0104]

[0105] In the formula: , , Defined as multi-scale pooling branch weights, and =1, The feature of global average pooling is defined by the following formula:

[0106]

[0107] For local pooling, the following formula is used:

[0108]

[0109] AvgPool refers to average pooling;

[0110] For edge feature pooling, the following formula is used:

[0111]

[0112] In the above three equations, H and W represent the height and width of the input image, respectively. Defined as the gradient operator.

[0113] S2.2.2 Color Perception Channel Grouping Unit: Based on the flame color gradient characteristics, the channels are grouped according to color sensitivity using the following grouping formula:

[0114]

[0115] in Defined as color group weight, The channel weight vector output after excitation operation for color grouping g. , , For the weights of the fully connected layer, Assigning weights to each color group, in the definition of the parameters above, g∈{red (red group), yellow (yellow group), bright (bright group)};

[0116] This is the vector after color-sensitive compression. It is the ReLU activation function. Use the Sigmoid activation function;

[0117] S2.2.3, Spatiotemporal Joint Attention Unit: Processes consecutive frames to address the dynamic characteristics of flames.

[0118]

[0119]

[0120] In the above formula, Defined as the compression feature of the c-th channel across consecutive frames. Defined as the spatiotemporal attention weights of the final output, with dimensions [C, 1, 1] and values ​​ranging from [0, 1]. The number of frames before time t in the feature map, i.e. This indicates that channel C at time point t- Coordinates in the input feature map Location, , These are feature maps that enhance attention in both time and space, where... Indicates concatenation of channel dimensions;

[0121] S2.2.4. For small-sized non-compliant items, introduce a spatial importance diagram:

[0122]

[0123] Weighted enhancement is applied to small target areas:

[0124]

[0125] For position ( The spatial importance value at position () reflects the salience of that position in the feature map, where C is the total number of channels in the feature map. This is the enhanced feature map of the c-th channel. The coordinates of the feature map input for the c channel are: The position is given by H and W, which represent the height and width of the feature map.

[0126] S2.2.5, For transparent / reflective objects:

[0127]

[0128]

[0129] Defined as an edge feature map, Sobel is the Sobel operator, an image processing filter commonly used for edge detection.

[0130] The weight of the c-th channel after edge-guided enhancement. represents the weight of the c-th channel in the original attention mechanism. This is the edge enhancement coefficient, used to control the degree of influence of edge information on the weights.

[0131] S2.3, Recalibration operation: Multiply the learned weights by the original feature map:

[0132]

[0133] in, This is the enhanced feature map of the c-th channel. Let c be the weight of the c-th channel. This is the original feature map of the c-th channel.

[0134] S3. Model Training and Optimization: Using the training set obtained in step S1 Train the constructed neural network model; use the training set Input the data into the network in batches and minimize the loss function. for:

[0135]

[0136] in, The weights are the bounding box loss. Weight the target confidence level loss. The weights are used for classification loss.

[0137] The following CIoU formula is used to optimize the proximity of the output bounding box to the actual bounding box in terms of position and size:

[0138]

[0139] In the formula, The distance is represented by the center point, and l is the length of the diagonal of the minimum bounding box. For dynamic weighting coefficients, ,in ;

[0140] W and h are the width and height of the predicted bounding box. , These are the actual frame width and height, respectively.

[0141] Intersection over Union (IoU) represents the degree of overlap between the predicted bounding box and the manually annotated region, and is expressed by the following formula:

[0142]

[0143] B, These are the predicted bounding box and the ground truth bounding box, respectively.

[0144] The target confidence loss value is used to optimize the model's ability to determine whether the target exists or not. A binary cross-entropy loss function is employed.

[0145]

[0146] N is the total number of predicted locations. For the first The actual situation at each location, with values ​​of 0 and 1 for no target and with target, respectively. For the model to the first The probability of a location prediction.

[0147] To optimize the model's ability to identify target categories, a multi-class cross-entropy loss function is used:

[0148]

[0149] N is the total target number. Let q be the category of the i-th target, and let the true category q take values ​​of 1 and 0 for the other categories. Here, O represents the probability predicted by the model, O is the total number of categories, and o represents the o-th category.

[0150] S4. Deploy the trained model into the real-time detection system to identify and warn of dangerous sources and violations in the input video stream in real time.

[0151] Example 2

[0152] The method provided in this application mainly includes three stages: data preparation, model training, and system application.

[0153] First, the dataset was constructed (corresponding to step S1). Some images were downloaded from the platform. Simultaneously, relevant laboratory videos were obtained from relevant websites, and one image was extracted every 30 frames. Five violations were defined: insufficient protective gear, eating, drinking, using a mobile phone, and sleeping, and are further specified in Table 1.

[0154] Table 1 Reference Table for Laboratory Violations

[0155]

[0156] The objects to be detected are items related to these behaviors, such as gloves, cutlery, and mobile phones. Images of flames and behaviors of the same pixel size and format were collected. All images were manually labeled using LabelImg software. After labeling, each object was assigned its category and location information. This information was saved as a text file for model training.

[0157] Then, model training is performed (corresponding to step S2). Figure 1 This is a schematic diagram illustrating the working principle of the FSE attention mechanism, where:

[0158] The backbone network is used to extract basic features from the input image. In the diagram, "SE network" represents a convolutional layer module with an improved FSE attention mechanism. The FSE module is inserted at various stages of the backbone network, enhancing feature representation capabilities through squeezing and activation operations. Squeezing corresponds to global average pooling, and activation corresponds to the weight learning operation of the fully connected layers.

[0159] The neck network connects the backbone network and the detection head, and is used for feature fusion and enhancement. This embodiment employs a feature pyramid structure to fuse features from different levels, thereby improving the detection capability of multi-scale targets.

[0160] The head network is responsible for the final object detection task. In this embodiment, the head network contains three detection heads, corresponding to object detection at large, medium, and small scales, respectively. Each detection head outputs a detection box containing the bounding box location, width and height, confidence score, and category.

[0161] The post-processing module performs post-processing on the original detection boxes output by the detection head, mainly including non-maximum suppression, which is used to remove redundant overlapping detection boxes and retain the detection result with the highest confidence as the final output.

[0162] An FSE attention module was added to the backbone network of this model. The structure of the FSE module is as follows: Figure 2As shown. It can automatically learn the importance of each channel feature. The model is trained using the pre-built dataset. Parameters such as training epochs and learning rate are set. During training, the model continuously adjusts its internal parameters to reduce prediction error. Two models are trained: one is the original model, and the other is a model with the FSE module added.

[0163] When constructing the original model, the input image I and the convolution kernel h are used to output the feature map F, and its operation is performed using the following formula:

[0164]

[0165] The coordinates are inside the convolution kernel. This is the input channel number. Here, k represents the position coordinates of the input image, and k represents the output channel number. This represents the total number of input channels. , These represent the height and width of the convolution kernel, respectively.

[0166] In the improved attention mechanism, the computation process of the FSE module includes:

[0167] S2.2.1 Multi-scale feature sensing unit: A multi-branch pooling strategy is adopted to address the various characteristics of flames.

[0168]

[0169] In the formula: , , Defined as multi-scale pooling branch weights. , The feature of global average pooling is defined by the following formula:

[0170]

[0171] Input the coordinates on the image for channel c.

[0172] For local pooling, the following formula is used:

[0173]

[0174] For edge feature pooling, the following formula is used:

[0175]

[0176] Input the coordinates on the image for channel c.

[0177] In the above three equations, H and W represent the height and width of the input image, respectively.

[0178] S2.2.2 Color Perception Channel Grouping Unit: Based on the flame color gradient characteristics, the channels are grouped according to color sensitivity using the following grouping formula:

[0179]

[0180] in Defined as color group weight, The channel weight vector output after the excitation operation for color group g. , , For the weights of the fully connected layer, g∈{red group, yellow group, bright group};

[0181] This is the vector after color-sensitive compression. It is the ReLU activation function. This is the Sigmoid activation function.

[0182] S2.2.3, Spatiotemporal Joint Attention Unit: Processes consecutive frames to address the dynamic characteristics of flames.

[0183]

[0184]

[0185] In the above formula, Defined as the compression feature of the c-th channel across consecutive frames. Defined as the spatiotemporal attention weights of the final output, with dimensions [C, 1, 1] and values ​​ranging from [0, 1]. The number of frames before time t in the feature map, i.e. This indicates that channel C at time point t- Coordinates in the input feature map Location, , These are feature maps that enhance attention in both time and space;

[0186] in This indicates concatenation of channel dimensions.

[0187] S2.2.4. For small-sized non-compliant items, introduce a spatial importance diagram:

[0188]

[0189] This formula represents the average of the absolute values ​​of the feature maps from all channels to obtain the spatial importance map. , The larger the value, the higher the position. The more prominent a feature is in the feature map, the more likely it is to be the region where a small target is located.

[0190] Weighted pooling of the original feature map is performed using the spatial importance map:

[0191]

[0192] S2.2.5. For transparent / reflective objects with sharp edges, an edge enhancement guiding unit is used:

[0193]

[0194] Sobel refers to the Sobel operator;

[0195] Incorporate edge features into channel weights:

[0196]

[0197] Defined as the edge feature map of the c channel. The weight of the c-th channel after edge-guided enhancement. This represents the weight of the c-th channel in the original attention mechanism; This is the edge enhancement coefficient, used to control the degree of influence of edge information on the weights.

[0198] After training, evaluate model performance using metrics such as precision, recall, and mean precision (AP / mAP). Figure 3 As shown, for flame detection, the average precision (AP) of the original model is 0.959, while the AP of the model combined with SE is 0.966. For violation detection, the mAP of the original model is 0.809, while the mAP of the model combined with the SE module is 0.839. Meanwhile, the AP values ​​of the FSE module's multi-scale FSE, spatiotemporal FSE, color grouping FSE, and small target FSE are 0.972, 0.981, 0.978, and 0.975, respectively, with mAP values ​​of 0.851, 0.847, 0.843, and 0.868. Experiments show that adding the FSE module improves model accuracy compared to the original model and simply adding the SE module. The model evaluation index formula is as follows:

[0199] Accuracy:

[0200]

[0201] Recall rate:

[0202]

[0203] Average accuracy:

[0204]

[0205] Multi-class average precision:

[0206]

[0207] N represents the number of detection categories.

[0208] Next, the application system is built. When using the system, users first select the video, image, or turn on the camera to be analyzed. Then, the system calls the trained model to identify each frame of the image. Identified flames and violations are marked with different colored boxes, and category labels and confidence levels are displayed. Users can see real-time statistics on the interface. The system can also save alarm information and results for later review.

[0209] Figure 4 The system's detection performance is demonstrated. It can be seen that the system can identify whether a person is wearing goggles, using a mobile phone, or engaging in other behaviors in complex backgrounds.

[0210] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in this application, based on the technical solution and patent concept of this application, should be included within the scope of protection of this application.

Claims

1. A method for identifying hazards and violations based on deep learning, characterized in that, Includes the following steps: S1. Dataset Construction: Randomly sample frames to collect images containing flames and violations by laboratory personnel. Label the flames and defined violations in the images. Combine measurement sequence i with image I to form a 3D vector, which constitutes the dataset. Use RGB three-channel format as the input to the neural network. Divide the dataset proportionally into a training set. and verification set ; S2. Constructing the detection model: Constructing a target detection neural network, and setting an improved FSE attention mechanism module in the network, using ReLU and Sigmoid activation functions; the neural network output is a set containing multiple detection boxes; specifically including the following sub-steps: S2.1 When constructing the original model, the input image I and the convolution kernel h are used to output the feature map F, and the operation is performed using the following formula; ； in, , The coordinates inside the convolution kernel, with values ​​ranging from 0 ≤ m < 0. , 0 ≤ n < , This is the index of the input channel. , Let k be the position coordinates of the input image I, and k be the output channel number. This represents the total number of input channels. , These are the height and width of the convolution kernel, respectively; S2.2, The improved FSE attention mechanism module performs a squeezing operation on the input feature map F. The global average pooling result for the c-th channel is: ； in, Let H be the compressed feature of the c-th channel, where H and W are the height and width of the feature map, and C is the total channel size. , Input the coordinate values ​​on the feature map for the c channel, i.e. c-channel input feature map The pixel value in the middle coordinate is point; S2.3, Activation Operation: Through two fully connected layers, channel weights are learned. ； z is the compressed vector. This is the channel weight vector output after the excitation operation; , For the weights of the fully connected layer, It is the ReLU activation function. Use the Sigmoid activation function; S2.4, Recalibration operation: Multiply the learned weights by the feature map. ； in, This is the enhanced feature map of the c-th channel. Let c be the weight of the c-th channel. Input the feature map for channel c; S3. Model Training and Optimization: Using the training set obtained in step S1 Train the constructed neural network model; use the training set Input the data into the network in batches and minimize the loss function. for: ； in, The weights are the bounding box loss. Weight the target confidence level loss. For classification loss weights; The following CIoU formula is used to optimize the proximity of the output bounding box to the actual bounding box in terms of position and size: ； In the formula, The distance is represented by the center point, and l is the length of the diagonal of the minimum bounding box. For dynamic weighting coefficients, ,in: ； W and h are the width and height of the predicted bounding box. , These are the actual frame width and height, respectively; Intersection over Union (IoU) represents the degree of overlap between the predicted bounding box and the manually annotated region, and is expressed by the following formula: ； B, These are the predicted bounding box and the ground truth bounding box, respectively. To optimize the model's ability to determine whether a target exists, a binary cross-entropy loss function is used: ； N is the total number of predicted locations; For the first The actual situation at each location, with a value of 0 for no target and a value of 1 for a target; For the model to the first The probability of predicting a location; To optimize the model's ability to identify target categories, a multi-class cross-entropy loss function is used: ； N is the total target number; For the first The target is categorized into categories, with the true category q having a value of 1 and the non-true category having a value of 0; The probability predicted by the model is O, where O is the total number of categories and o represents the o-th category. S4. Deploy the trained model to the real-time detection system to identify and warn of dangerous sources and violations in the input video stream in real time; The improved FSE attention mechanism module includes: (1) Multi-scale feature perception unit, which adopts multi-branch fusion of global pooling, local pooling and edge pooling; (2) Color perception channel grouping unit, which groups the feature channels according to color sensitivity; (3) Spatiotemporal joint attention unit, which processes the temporal dynamic features of continuous frame sequences; (4) Small target spatial recalibration units are used to weight the small target regions using a spatial importance map; Alternatively, it may include edge-enhancing guidance units to improve the recognition of transparent or reflective objects.

2. The method for identifying hazards and violations based on deep learning according to claim 1, characterized in that, In step S1, the resolution and format of the image are consistent, and the pixel values ​​are normalized to the range [0,1].

3. The method for identifying hazards and violations based on deep learning according to claim 1, characterized in that, In step S1, the random frame sampling specifically involves extracting image frames from the video at fixed intervals of N frames, where N is 30.

4. The method for identifying hazards and violations based on deep learning according to claim 1, characterized in that, In step S1, the personnel violations include not wearing protective equipment, eating, drinking, playing on mobile phones, and sleeping.

5. The method for identifying hazards and violations based on deep learning according to claim 1, characterized in that, In step S2, each detection box has four parameters: bounding box position, width and height, confidence score, and class.

6. The method for identifying hazards and violations based on deep learning according to claim 1, characterized in that, The output of the multi-scale feature sensing unit is calculated as follows: ； In the formula: , , Defined as multi-scale pooling branch weights, and =1, Global average pooling uses the following formula: ； For local pooling, the following formula is used: ； For edge feature pooling, the following formula is used: 。 7. The method for identifying hazards and violations based on deep learning according to claim 1, characterized in that, The small target spatial recalibration unit specifically includes constructing a spatial importance graph: ； Weighted enhancement is applied to small target areas: ； For feature map location The spatial importance value reflects the saliency of that location in the feature map, where C is the total number of channels in the feature map. This is the enhanced feature map of the c-th channel.

8. The method for identifying hazards and violations based on deep learning according to claim 1, characterized in that, The edge enhancement guidance unit specifically includes extracting edge feature maps; ； in Defined as an edge feature map, Sobel refers to the Sobel operator; Incorporate edge features into channel weights: ； Defined as the edge feature map of the c channel. The weight of the c-th channel after edge-guided enhancement. This represents the weight of the c-th channel in the original attention mechanism; This is the edge enhancement coefficient, used to control the degree of influence of edge information on the weights.

Images