Method for quickly judging violation operating personnel based on image recognition
By applying image recognition technology in the power production and operation and maintenance site, using improved space pyramid network and hybrid attention convolution network to identify safety helmet numbers, the problem of rapid judgment by illegal operators on the power production and operation and maintenance site is solved. , achieving high accuracy and high efficiency identification of illegal operators, reducing safety hazards.
Patent Information
- Application Number
- CN202410953126.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-22
AI Technical Summary
At the power production and operation and maintenance site, it is difficult to quickly and accurately identify and determine illegal operators in existing technologies, resulting in greater safety risks.
Using a pseudo-twin network framework based on image recognition technology, through hard helmet number identification, we build an improved spatial pyramid network and a hybrid attention convolution network, combining attention mechanisms and channel attention mechanisms to realize hard helmet detection and number identification, so as to quickly Determine the illegal operation personnel.
It improves the accuracy and efficiency of identification of illegal operators, reduces the probability of power accidents and casualties, and achieves accurate and rapid judgment of illegal operators.
Smart Images

Figure CN119007097A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of power production and operation and maintenance, and in particular to a method for quickly determining illegal operators in a power station based on image processing technology. Background Art
[0002] Illegal operation by operators at power production and operation sites is the main cause of safety accidents. Illegal operation accidents are almost always caused by operators with weak skills, low professional level, weak safety awareness, and arbitrary execution of operation tasks, failure to follow rules and regulations, and inadequate safety measures. At present, relevant technologies have been applied to power production and operation sites to detect illegal operation behaviors of operators. However, relevant methods and applications for rapid determination of illegal operators have not yet been seen. Determination of illegal operators is a vital task in the power production industry. If targeted safety education and training is not carried out in a timely manner for illegal operators, it will be a huge safety hazard for power production and operation.
[0003] In the task of personnel detection and judgment, pedestrian re-identification detection and face detection are relatively well-known and maturely applied. In the task of judging illegal workers, the illegal workers can be judged with the help of face recognition methods. In addition, the illegal workers can also be judged with the help of the numbers on the employees' safety helmets. Currently in the Three Gorges Power Plant, each employee's safety helmet has a specific number; in addition, the colors of safety helmets in different departments are also different, including red, yellow, and blue. Taking into account that on-site workers often turn their backs to the camera to perform related operations and maintenance tasks, the present invention mainly judges illegal workers from the perspective of safety helmet number re-identification.
[0004] Thanks to the rapid development of computer vision technology, image processing technology and pattern recognition technology, major breakthroughs have been made in image recognition and character recognition, and some of the results have been widely used in actual production and life (such as handwritten character recognition, license plate recognition, etc.). From the perspective of helmet number re-identification, the present invention proposes a method for quickly determining illegal operators based on image recognition, and designs a pseudo-twin network framework to identify illegal operators. A method for quickly determining illegal operators is provided for the Three Gorges Power Plant, so that the power plant can provide targeted safety education and training for illegal operators, reducing the probability of power accidents and casualties. Summary of the invention
[0005] The purpose of the present invention is to identify the illegal workers from the perspective of helmet number re-identification, so as to improve the identification accuracy and efficiency of illegal workers in power plants, and to propose a method for quickly identifying illegal workers based on image recognition.
[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0007] A method for quickly determining a person who violates regulations based on image recognition comprises the following steps:
[0008] Step 1, creating a standard image dataset A of the back of employee helmets, including collecting standard images of the back of all employee helmets in the target area, and performing corresponding preprocessing to obtain dataset A;
[0009] Step 2, creating a back image dataset B of employees wearing helmets at work;
[0010] Step 3: Construct a pseudo-twin network framework as a model for determining illegal operators;
[0011] Step 4: Divide the data set B into a training set, a validation set, and a test set according to a certain ratio, and train and validate the model for determining illegal workers;
[0012] Step 5: Deploy and call the illegal operation personnel determination model.
[0013] In step 2, the following steps are included:
[0014] Step 2.1, collecting back video images of all employees in the target area wearing safety helmets while working in different working scenarios;
[0015] Step 2.2, frame and crop the collected employee back video data to obtain a multi-scale illegal operation personnel dataset B;
[0016] Step 2.3: perform category and semantic annotation on dataset B:
[0017] 1) Create a category label for each employee in dataset B (represented by 0, 1, 2, 3...M, where M represents the total number of employees in the target area). That is, images with the same helmet number have the same label; images with different helmet numbers have different labels.
[0018] 2) Use LabelMe to semantically annotate the images in dataset B to obtain a single-channel binary image, so that each pixel value represents the integer identifier of the category, and use 0 to identify the non-helmet area and 1 to represent the helmet area.
[0019] In step 3, the constructed pseudo twin network framework specifically includes:
[0020] 1) Construct an improved spatial pyramid network as one of the branches of the pseudo-twin network framework for helmet detection and number recognition of illegal workers in work scenarios;
[0021] The improved spatial pyramid network branch consists of input (back image of employees wearing helmets), image initialization layer, input layer convolution module, attention mechanism pyramid pooling module, output layer convolution module, and mixed attention convolution module;
[0022] The image initialization layer is mainly used to preprocess the input image (the back image of the employee wearing a helmet at work) and scale the image to a certain size;
[0023] The input layer convolution module serves as the backbone feature extraction network; this module uses the main structure of MobileNetV2 (1 convolution layer and 7 bottleneck layers) to implement feature extraction;
[0024] The attention mechanism pyramid pooling module is used to aggregate contextual information from different regions and improve the ability to obtain global information. The module first divides the input feature map into grids at four scales, then performs a channel attention mechanism operation on each grid of the feature map to obtain an enhanced feature map, and then forms a final feature representation containing local and global contextual information through upsampling and concatenation operations.
[0025] The output layer convolution module is mainly used to classify the feature map at the pixel level and output the final semantic segmentation result, with the purpose of segmenting the helmet area. The module consists of a 3×3 convolution layer, a 1×1 convolution layer and an upsampling layer. The 3×3 convolution layer is used to integrate the features, and the 1×1 convolution layer is used to adjust the channels (adjusted to two channels, one channel for the semantic segmentation result of the helmet and the other channel for the semantic segmentation result of the non-helmet area). The width and height of the final output layer are made consistent with the input image through upsampling.
[0026] The hybrid attention convolution module is mainly used to learn the features of helmet numbers and complete number recognition; the module consists of 1 logical AND operation, 1 image cropping layer, 1 1×1 convolution layer, 1 hybrid attention mechanism module, 1 3×3 convolution layer, 1 fully connected layer and 1 Softmax layer; the logical AND operation is used to filter the non-helmet area, and the image cropping layer is used to crop the helmet area and use it as the input image for helmet number extraction;
[0027] 2) Construct a hybrid attention convolutional network as another branch of the pseudo-twin network framework for number recognition of the standard image on the back of the helmet; the hybrid attention convolutional network of this branch is mainly composed of 1 image initialization layer, 1 1×1 convolutional layer, 3 BC structures (the proposed BC structure is 1 bottleneck layer and 1 hybrid attention mechanism module), 1 3×3 convolutional layer, 2 fully connected layers and 1 Softmax layer; the image initialization layer is mainly used to preprocess the input image (standard image of the helmet) and scale the image to a certain size.
[0028] In step 3, the structure of the improved spatial pyramid network is as follows: the output of the first image initialization layer is connected to the input of the input layer convolution module, the output of the input layer convolution module is connected to the input of the attention mechanism pyramid pooling module, the output of the attention mechanism pyramid pooling module is connected to the input of the output layer convolution module, and the output of the output layer convolution module is connected to the input of the mixed attention convolution module; the input of the first image initialization layer is used to input the back image of the employee wearing a safety helmet when working;
[0029] In step 3, the structure of the hybrid attention convolutional network is: the output of the second image initialization layer is connected to the input of the 1×1 convolutional layer, the output of the 1×1 convolutional layer is connected to the input of the first BC structure, the output of the first BC structure is connected to the input of the second BC structure, the output of the second BC structure is connected to the input of the third BC structure, the output of the third BC structure is connected to the input of the 3×3 convolutional layer, the output of the 3×3 convolutional layer is connected to the input of the first fully connected layer, and the output of the first fully connected layer is connected to the input of the second fully connected layer.
[0030] The attention mechanism pyramid pooling module is specifically:
[0031] With x∈R H×W×C Represents the input feature map of the module, where H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels of the feature map;
[0032] Divide x into 1×1, 2×2, 3×3, and 7×7 grids, and obtain four feature maps after grid division: x1, x2, x3, and x7. Among them, x1 represents the feature map of x as a subregion, x2 represents the feature map of x divided into 2×2 subregions, x3 represents the feature map of x divided into 3×3 subregions, and x7 represents the feature map of x divided into 7×7 subregions;
[0033] Use x ij Represents the feature map x after grid division i The j-th sub-region of , where, when i=1, j=1; when i=2, j={1,2,3,4}; when i=3, j={1,2,3,4,5,6,7,8,9}; when i=7, j={1,2,3,…,48,49};
[0034] x ij Perform global average pooling and global maximum pooling operations to obtain two vectors of size 1×1×C:
[0035]
[0036] Among them, AvgPool(·) represents the average pooling operation, MaxPool(·) represents the maximum pooling operation, represents the output of average pooling, represents the output of the maximum pooling;
[0037] Will and They are input into the shared multi-layer perceptron (MLP) respectively, and the results are added together and the channel attention map is obtained through the sigmoid activation function:
[0038]
[0039] Where σ represents the sigmoid activation function, MLP(·) represents a multi-layer perceptron network with an output layer size of C;
[0040] Attention pooling feature map M for each channel i Bilinear interpolation is performed to achieve upsampling to restore it to the size of the input feature map of the module, and all upsampled feature maps are concatenated with the input feature map in the channel dimension to form a new feature map. It is further processed through a convolution layer to reduce the number of channels and fuse features.
[0041] In step 4, the illegal operation personnel identification model is trained and verified as follows:
[0042] Step 4.1, set the hyperparameters required for training, including epoch, batch_size, learning rate, training optimizer, and loss function; the loss function is the weighted sum of contrast loss, cross entropy loss, and Dice loss:
[0043] The contrastive loss function used to train the pseudo-Siamese network can be expressed as follows:
[0044]
[0045] Among them, y represents the characteristic function, that is, m is the interval parameter, which is used to ensure that the two samples (B i ,A j ) is greater than m;
[0046] The Dice loss function used to measure the semantic segmentation results of the helmet area is expressed as follows:
[0047]
[0048] Among them, for the segmentation task, X and Y represent the semantic segmentation label and semantic segmentation prediction mask respectively, |X∩Y| represents the intersection of the X and Y sets, and |X| and |Y| represent the number of their elements;
[0049] The cross entropy loss function used to measure the helmet number recognition results is expressed as follows:
[0050]
[0051] Among them, z i Refers to the probability distribution of the possible violating employee labels output by the violating worker judgment model, l i is the one-hot form of the sample label of the i-th sample, N = batch_size;
[0052] Then the loss function of the pseudo twin network framework can be expressed as:
[0053] L=αL cross +βL Adv +λL Dice :
[0054] Among them, α, β, and λ represent weights, α+β+λ=1;
[0055] Step 4.2, set the model evaluation indicators, the evaluation indicators are accuracy and F1-score;
[0056] Step 4.3, initialize network parameters;
[0057] Step 4.4, input the training set in data set B and data set A into the constructed illegal operation personnel judgment model (wherein, the training set in data set B is used as the input of the improved spatial pyramid network branch, and data set A is used as the input of the hybrid attention convolutional network branch), start the training process, and update the model parameters through the gradient descent algorithm according to the calculated loss until the training is completed;
[0058] Step 4.5, freeze the model parameters, input the validation set in dataset B and dataset A into the trained illegal operator judgment model (wherein, the validation set in dataset B is used as the input of the improved spatial pyramid network branch, and dataset A is used as the input of the hybrid attention convolutional network branch), optimize and adjust the model through evaluation indicators, and finally, input the test set in dataset B and dataset A into the optimized illegal operator judgment model (wherein, the test set in dataset B is used as the input of the improved spatial pyramid network branch, and dataset A is used as the input of the hybrid attention convolutional network branch) to evaluate the performance of the model.
[0059] In step 5, the deployment and calling of the illegal operation personnel determination model adopts the following steps:
[0060] Step 5.1, according to the preset evaluation indicators (Accuarcy and F1-score), determine the model with the best evaluation indicators in the verification process; call the torch.save() function to save the parameters of the optimal model into the .pth file, and save the model architecture into the .py file.
[0061] Step 5.2: In the operation area monitoring system, use the Python import function to import the saved optimal model, and use the load_state_dict() function to load the saved optimal model parameters to implement the deployment of the optimal model in the monitoring system.
[0062] Step 5.3, store data set A in the data call path of the deployment model;
[0063] In step 5.4, when an operator is found to have violated traffic rules (based on the violation behavior detection model), the image of the violating operator is automatically input into the improved spatial pyramid network branch of the violating operator judgment model (the model will automatically call the image in dataset A and use it as the input of the hybrid attention convolutional network branch), and the violating operator is judged in real time, and the judgment result is returned.
[0064] Compared with the prior art, the present invention has the following technical effects:
[0065] 1) Based on image recognition technology, the present invention designs a rapid identification model for illegal workers based on a pseudo-twin network framework from the perspective of helmet number re-identification, thereby realizing rapid identification of illegal workers;
[0066] 2) The present invention uses the spatial pyramid network to detect the helmets of illegal workers, and improves its internal structure pyramid pooling module. Specifically, the average pooling operation in the pyramid pooling module is improved to a channel attention mechanism operation, thereby optimizing the feature extraction process and enhancing the model's attention to important channels of the feature map;
[0067] 3) The contrast loss of the pseudo-twin network in the present invention is used to promote learning more similar features from different images of the same employee (the back image of the illegal worker and the standard image of the back of the helmet). With the help of the established number of the helmet, this detection method can achieve accurate and rapid determination of illegal workers. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:
[0069] Figure 1 is a flow chart of the present invention;
[0070] Figure 2 This is an example of a standard image of the back of a helmet in data set A of the present invention;
[0071] Figure 3 This is a pseudo twin network framework diagram in the present invention;
[0072] Figure 4 Schematic diagram of the pyramid pooling module of the attention mechanism in the present invention. DETAILED DESCRIPTION
[0073] like Figure 1 As shown, this embodiment provides a method for quickly determining a person who violates regulations based on image recognition, including the following steps:
[0074] Step 1, creating a standard image dataset A of the back of employee helmets, including the following steps:
[0075] Step 1.1, put the safety helmets of all employees in the target area in the designated position and take photos to obtain the original standard image set of safety helmets with a resolution of 1920 pixels × 1080 pixels;
[0076] Step 1.2: Use the image cropping tool PicResize to crop the helmet from the original standard image of the helmet to obtain the standard image dataset A of the back of the helmet.
[0077] Step 2, creating a back image dataset B of employees wearing helmets at work, including the following steps:
[0078] Step 2.1, using the camera installed in the work scene to collect back video images of all employees wearing safety helmets while working in different areas;
[0079] Step 2.2, use the VideoCapture function in the OpenCV extension library cv2 to frame the collected back video images of employees wearing safety helmets while working. For each employee, select 10 images from the video frames obtained from the same camera. Then use the image cropping tool PicResize to crop the employee from the image. Obtain a multi-scale illegal operation personnel dataset B;
[0080] Step 2.3: perform category and semantic annotation on dataset B:
[0081] 1) Use Excel to manually create a category label for each employee in data set B (represented by 0, 1, 2, 3...M-1, where M represents the total number of employees in the target area). That is, images with the same helmet number have the same label; images with different helmet numbers have different labels.
[0082] 2) Use LabelMe to semantically annotate the images in dataset B to obtain a single-channel binary image, so that each pixel value represents the integer identifier of the category, and use 0 to identify the non-helmet area and 1 to represent the helmet area.
[0083] Step 3: Construct a pseudo-twin network framework as a model for determining illegal operators; Figure 3 As shown;
[0084] 1) Construct an improved spatial pyramid network as one of the branches of the pseudo-twin network framework for helmet detection and number recognition of illegal workers in work scenarios. The improved spatial pyramid network branch consists of input (back image of employees wearing helmets at work), image initialization layer, input layer convolution module, attention mechanism pyramid pooling module, output layer convolution module, and mixed attention convolution module.
[0085] The image initialization layer is mainly used to preprocess the input image (the back image of the employee wearing a safety helmet at work). The Padding function is used to adjust the input image to a square size, and then it is scaled proportionally to an image of 224 pixels × 224 pixels with a depth of 3.
[0086] The input layer convolution module serves as the backbone feature extraction network; this module uses the main structure of MobileNetV2 (1 convolution layer and 7 bottleneck layers) to perform deep feature extraction on the input image of size 224×224×3, and obtains a feature map of size 7×7×160.
[0087] The attention mechanism pyramid pooling module is used to aggregate contextual information from different regions and improve the ability to obtain global information. The module first divides the input feature map of size 7×7×160 into four scales (the four grids are 1×1, 2×2, 3×3, and 7×7), and then performs a channel attention mechanism operation on each grid of the feature map to obtain four enhanced feature maps of different sizes: 1×1×160, 2×2×160, 3×3×160, and 7×7×160. After upsampling and concatenation operations, a feature layer of size 7×7×640 containing local and global contextual information is formed.
[0088] The output layer convolution module is mainly used to classify the feature map at the pixel level and output the final semantic segmentation result, with the purpose of segmenting the helmet area. This module consists of a 3×3 convolution layer, a 1×1 convolution layer and an upsampling layer. The 3×3 convolution layer is used to integrate the features, and the 1×1 convolution layer is used to adjust the channels (adjusted to two channels, one channel for the semantic segmentation result of the helmet, and the other channel for the semantic segmentation result of the non-helmet area). The width and height of the final output layer are consistent with the input image through upsampling.
[0089] The hybrid attention convolution module is mainly used to learn the features of helmet numbers and complete number recognition; the module consists of 1 logical AND operation, 1 image cropping layer, 1 1×1 convolution layer, 1 hybrid attention mechanism module, 1 3×3 convolution layer, 1 fully connected layer and 1 Softmax layer; the logical AND operation is used to filter the non-helmet area, and the image cropping layer is used to crop the helmet area and use it as the input image for helmet number extraction.
[0090] 2) Construct a hybrid attention convolutional network as another branch of the pseudo twin network framework for serial number recognition of the standard image on the back of the helmet. The hybrid attention convolutional network of this branch mainly consists of 1 image initialization layer, 1 1×1 convolutional layer, 3 BC structures (the proposed BC structure is 1 bottleneck layer and 1 hybrid attention mechanism module), 1 3×3 convolutional layer, 2 fully connected layers and 1 Softmax layer. The image initialization layer is mainly used to preprocess the input image (standard image of the helmet), adjust the input image to a square size using the Padding function, and then scale it geometrically to an image of 640 pixels × 640 pixels with a depth of 3.
[0091] Preferably, the attention mechanism pyramid pooling module of the spatial pyramid network is improved, specifically as follows:
[0092] With x∈R H×W×C Represents the input feature map of the module. H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels of the feature map. Divide x into 1×1, 2×2, 3×3, and 7×7 grids to obtain four feature maps after grid division: x1, x2, x3, and x7. Among them, x1 represents the feature map of x as a subregion, x2 represents the feature map of x divided into 2×2 subregions, x3 represents the feature map of x divided into 3×3 subregions, and x7 represents the feature map of x divided into 7×7 subregions.
[0093] Use x ij Represents the feature map x after grid division i The jth sub-region of . When i=1, j=1; when i=2, j={1,2,3,4}; when i=3, j={1,2,3,4,5,6,7,8,9}; when i=7, j={1,2,3,…,48,49}. ij Perform global average pooling and global maximum pooling operations to obtain two vectors of size 1×1×C:
[0094]
[0095] Among them, AvgPool(·) represents the average pooling operation, MaxPool(·) represents the maximum pooling operation, represents the output of average pooling, Represents the output of max pooling.
[0096] Will and They are input into the shared multi-layer perceptron (MLP) respectively, and the results are added together and the channel attention map is obtained through the sigmoid activation function:
[0097]
[0098] Where σ represents the sigmoid activation function and MLP(·) represents a multi-layer perceptron network with an output layer size of C.
[0099] Attention pooling feature map M for each channel i Bilinear interpolation is performed to achieve upsampling to restore it to the size of the input feature map of the module. All upsampled feature maps are concatenated with the input feature map in the channel dimension to form a new feature map. It is further processed through a convolutional layer to reduce the number of channels and fuse features.
[0100] Step 4: Divide the data set B into a training set, a validation set, and a test set in a ratio of 7:2:1, and train and validate the illegal operator judgment model, including the following steps:
[0101] Step 4.1, set the hyperparameters required for training, including setting epoch = 500, batch_size = 64, learning rate = 0.0001, training optimizer as Adam optimizer, and loss function. The loss function is the weighted sum of contrast loss, cross entropy loss, and Dice loss:
[0102] The contrastive loss function used to train the pseudo-Siamese network can be expressed as follows:
[0103]
[0104] Among them, y represents the characteristic function, that is, m is the interval parameter, which is used to ensure that the two samples (B i ,A j ) is greater than m.
[0105] The Dice loss function used to measure the semantic segmentation results of the helmet area is expressed as follows:
[0106]
[0107] Among them, for the segmentation task, X and Y represent the semantic segmentation label and semantic segmentation prediction mask respectively, |X∩Y| represents the intersection of the X and Y sets, and |X| and |Y| represent the number of their elements.
[0108] The cross entropy loss function used to measure the helmet number recognition results is expressed as follows:
[0109]
[0110] Among them, z i Refers to the probability distribution of the possible violating employee labels output by the violating worker judgment model, l i It is the one-hot form of the sample label of the i-th sample, N = batch_size.
[0111] Then the loss function of the pseudo twin network framework can be expressed as:
[0112] L=αL cross +βL Adv +λL Dice :
[0113] Among them, α, β, and λ represent weights, α+β+λ=1.
[0114] Step 4.2, set the model evaluation indicators, the evaluation indicators are accuracy and F1-score.
[0115] Step 4.3, initialize network parameters.
[0116] In step 4.4, the training set in dataset B and dataset A are input into the constructed illegal operation personnel judgment model (wherein the training set in dataset B is used as the input of the improved spatial pyramid network branch, and dataset A is used as the input of the hybrid attention convolutional network branch), and the training process is started. According to the calculated loss, the model parameters are updated through the gradient descent algorithm until the training is completed.
[0117] Step 4.5, freeze the model parameters, input the validation set in dataset B and dataset A into the trained illegal operator judgment model (wherein, the validation set in dataset B is used as the input of the improved spatial pyramid network branch, and dataset A is used as the input of the hybrid attention convolutional network branch), optimize and adjust the model through evaluation indicators, and finally, input the test set in dataset B and dataset A into the optimized illegal operator judgment model (wherein, the test set in dataset B is used as the input of the improved spatial pyramid network branch, and dataset A is used as the input of the hybrid attention convolutional network branch) to evaluate the performance of the model.
[0118] Step 5: Deploy and call the illegal operation personnel judgment model, including the following steps:
[0119] Step 5.1, according to the preset evaluation indicators (Accuarcy and F1-score), determine the model with the best evaluation indicators in the verification process; call the torch.save() function to save the parameters of the optimal model into the .pth file, and save the model architecture into the .py file.
[0120] Step 5.2: In the operation area monitoring system, use the Python import function to import the saved optimal model, and use the load_state_dict() function to load the saved optimal model parameters to implement the deployment of the optimal model in the monitoring system.
[0121] Step 5.3, store data set A in the data call path of the deployment model;
[0122] In step 5.4, when an operator is found to have violated traffic rules (based on the violation behavior detection model), the image of the violating operator is automatically input into the improved spatial pyramid network branch of the violating operator judgment model (the model will automatically call the image in dataset A and use it as the input of the hybrid attention convolutional network branch), and the violating operator is judged in real time, and the judgment result is returned.
[0123] The present invention constructs a pseudo-twin network framework with an improved spatial pyramid network and a hybrid attention convolutional network as branches to build a rapid discrimination model for illegal workers. First, the obtained illegal worker judgment samples are subjected to image preprocessing, and then the improved spatial pyramid network branch and the hybrid attention convolutional network branch are established under the pseudo-twin network framework. The improved spatial pyramid network is used to detect the safety helmets of illegal workers, which is used to separate the safety helmets from the background area well, and then realize the safety helmet number recognition; the hybrid attention convolutional network is used to learn the deep representation of the number on the standard image on the back of the safety helmet. The contrast loss of the pseudo-twin network is used to promote the learning of more similar features from different images of the same employee (the back image of the illegal worker and the standard image of the back of the safety helmet). With the help of the established number of the safety helmet, this detection method can realize the rapid judgment of illegal workers.
Claims
1. A method for quickly determining illegal workers based on image recognition, characterized in that: The following steps are involved: Step 1, creating a standard image dataset A of the back of employee helmets, including collecting standard images of the back of all employee helmets in the target area, and performing corresponding preprocessing to obtain dataset A; Step 2, creating a back image dataset B of employees wearing helmets at work; Step 3: Construct a pseudo-twin network framework as a model for determining illegal operators; Step 4: Divide the data set B into a training set, a validation set, and a test set according to a certain ratio, and train and validate the model for determining illegal workers; Step 5: Deploy and call the illegal operation personnel determination model.
2. The method according to claim 1, characterized in that In step 2, the following steps are included: Step 2.1, collecting back video images of all employees in the target area wearing safety helmets while working in different working scenarios; Step 2.2, frame and crop the collected employee back video data to obtain a multi-scale illegal operation personnel dataset B; Step 2.3, perform category and semantic annotation on dataset B: 1) Create a category label for each employee in dataset B, represented by 0, 1, 2, 3...M-1, where M represents the total number of employees in the target area. That is, images with the same helmet number have the same label, and images with different helmet numbers have different labels. 2) Use LabelMe to semantically annotate the images in dataset B to obtain a single-channel binary image, so that each pixel value represents the integer identifier of the category, and use 0 to identify the non-helmet area and 1 to represent the helmet area.
3. The method according to claim 1, characterized in that: In step 3, the constructed pseudo twin network framework specifically includes: 1) Construct an improved spatial pyramid network as one of the branches of the pseudo-twin network framework for helmet detection and number recognition of illegal workers in work scenarios; The improved spatial pyramid network branch consists of input, image initialization layer, input layer convolution module, attention mechanism pyramid pooling module, output layer convolution module, and mixed attention convolution module; The image initialization layer is mainly used to preprocess the input image and scale the image to a certain size; The input layer convolution module serves as the backbone feature extraction network; this module uses the main structure of MobileNetV2 to implement feature extraction; The attention mechanism pyramid pooling module is used to aggregate contextual information from different regions and improve the ability to obtain global information. The module first divides the input feature map into grids at four scales, then performs a channel attention mechanism operation on each grid of the feature map to obtain an enhanced feature map, and then forms a final feature representation containing local and global contextual information through upsampling and concatenation operations. The output layer convolution module is mainly used to classify the feature map at the pixel level and output the final semantic segmentation result, with the purpose of segmenting the helmet area. The module consists of a 3×3 convolution layer, a 1×1 convolution layer and an upsampling layer. The 3×3 convolution layer is used to integrate the features, and the 1×1 convolution layer is used to adjust the channels. The width and height of the final output layer are made consistent with the input image through upsampling. The hybrid attention convolution module is mainly used to learn the features of helmet numbers and complete number recognition; the module consists of 1 logical AND operation, 1 image cropping layer, 1 1×1 convolution layer, 1 hybrid attention mechanism module, 1 3×3 convolution layer, 1 fully connected layer and 1 Softmax layer; the logical AND operation is used to filter the non-helmet area, and the image cropping layer is used to crop the helmet area and use it as the input image for helmet number extraction; 2) Construct a hybrid attention convolutional network as another branch of the pseudo-twin network framework for number recognition of standard images on the back of helmets; the hybrid attention convolutional network of this branch is mainly composed of 1 image initialization layer, 1 1×1 convolutional layer, 3 BC structures, 1 3×3 convolutional layer, 2 fully connected layers and 1 Softmax layer; the image initialization layer is mainly used to preprocess the input image and scale the image to a certain size.
4. The method according to claim 3, characterized in that In step 3, the structure of the improved spatial pyramid network is as follows: the output of the first image initialization layer is connected to the input of the input layer convolution module, the output of the input layer convolution module is connected to the input of the attention mechanism pyramid pooling module, the output of the attention mechanism pyramid pooling module is connected to the input of the output layer convolution module, and the output of the output layer convolution module is connected to the input of the mixed attention convolution module; the input of the first image initialization layer is used to input the back image of the employee wearing a safety helmet when working; In step 3, the structure of the hybrid attention convolutional network is: the output of the second image initialization layer is connected to the input of the 1×1 convolutional layer, the output of the 1×1 convolutional layer is connected to the input of the first BC structure, the output of the first BC structure is connected to the input of the second BC structure, the output of the second BC structure is connected to the input of the third BC structure, the output of the third BC structure is connected to the input of the 3×3 convolutional layer, the output of the 3×3 convolutional layer is connected to the input of the first fully connected layer, and the output of the first fully connected layer is connected to the input of the second fully connected layer.
5. The method according to claim 3 or 4, characterized in that: The attention mechanism pyramid pooling module is specifically: With x∈R H×W×C Represents the input feature map of the module, where H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels of the feature map; Divide x into 1×1, 2×2, 3×3, and 7×7 grids, and obtain four feature maps after grid division: x1, x2, x3, and x7. Among them, x1 represents the feature map of x as a subregion, x2 represents the feature map of x divided into 2×2 subregions, x3 represents the feature map of x divided into 3×3 subregions, and x7 represents the feature map of x divided into 7×7 subregions; Use x ij Represents the feature map x after grid division i The j-th sub-region of , where, when i=1, j=1; when i=2, j={1,2,3,4}; when i=3, j={1,2,3,4,5,6,7,8,9}; when i=7, j={1,2,3,…,48,49}; x ij Perform global average pooling and global maximum pooling operations to obtain two vectors of size 1×1×C: Among them, AvgPool(·) represents the average pooling operation, MaxPool(·) represents the maximum pooling operation, represents the output of average pooling, represents the output of the maximum pooling; Will and They are input into the shared multi-layer perceptron (MLP) respectively, and the results are added together and the channel attention map is obtained through the sigmoid activation function: Where σ represents the sigmoid activation function, MLP(·) represents a multi-layer perceptron network with an output layer size of C; Attention pooling feature map M for each channel i Bilinear interpolation is performed to achieve upsampling to restore it to the size of the input feature map of the module, and all upsampled feature maps are concatenated with the input feature map in the channel dimension to form a new feature map; and further processed through a convolutional layer to reduce the number of channels and fuse features.
6. The method according to claim 1, characterized in that In step 4, the illegal operation personnel identification model is trained and verified as follows: Step 4.1, set the hyperparameters required for training, including epoch, batch_size, learning rate, training optimizer, and loss function; the loss function is the weighted sum of contrast loss, cross entropy loss, and Dice loss: The contrastive loss function used to train the pseudo-Siamese network can be expressed as follows: Among them, y represents the characteristic function, that is, m is the interval parameter, which is used to ensure that the two samples (B i ,A j ) is greater than m; The Dice loss function used to measure the semantic segmentation results of the helmet area is expressed as follows: Among them, for the segmentation task, X and Y represent the semantic segmentation label and semantic segmentation prediction mask respectively, |X∩Y| represents the intersection of the X and Y sets, and |X| and |Y| represent the number of their elements; The cross entropy loss function used to measure the helmet number recognition results is expressed as follows: Among them, z i Refers to the probability distribution of the possible violating employee labels output by the violating worker judgment model, l i is the one-hot form of the sample label of the i-th employee, N = batch_size; Then the loss function of the pseudo twin network framework can be expressed as: L=αL cross +βL Adv +λL Dice : Among them, α, β, and λ represent weights, α+β+λ=1; Step 4.2, set the model evaluation indicators, the evaluation indicators are accuracy and F1-score; Step 4.3, initialize network parameters; Step 4.4, input the training set in data set B and data set A into the constructed illegal operation personnel judgment model, start the training process, and update the model parameters through the gradient descent algorithm according to the calculated loss until the training is completed; Step 4.5, freeze the model parameters, input the validation set in dataset B and dataset A into the trained illegal operator judgment model, optimize and adjust the model through evaluation indicators, and finally input the test set in dataset B and dataset A into the optimized illegal operator judgment model to evaluate the performance of the model; Among them, the training set in dataset B is used as the input of the improved spatial pyramid network branch, and dataset A is used as the input of the hybrid attention convolutional network branch.
7. The method according to claim 2, characterized in that In step 5, the deployment and calling of the illegal operation personnel determination model adopts the following steps: Step 5.1, according to the preset evaluation indicators Accuarcy and F1-score, determine the model with the best evaluation indicators in the verification process; call the torch.save() function to save the parameters of the optimal model into the .pth file, and save the model architecture into the .py file; Step 5.2: In the operation area monitoring system, use the Python import function to import the saved optimal model, and use the load_state_dict() function to load the saved optimal model parameters to implement the deployment of the optimal model in the monitoring system. Step 5.3, store data set A in the data call path of the deployment model; In step 5.4, when an operator is found to have violated traffic rules, the image of the violating operator is automatically input into the improved spatial pyramid network branch of the violating operator judgment model. At this time, the model will automatically call the image in dataset A and use it as the input of the hybrid attention convolutional network branch to make real-time judgment on the violating operator and return the judgment result.
Citation Information
Patent Citations
Facial expression recognition method and device based on feature pyramid
CN113642505A
Crowd density estimation method based on spatial context learning network
CN116403152A
Breast ultrasonic tomographic image segmentation model establishment method and segmentation method
CN116433586A
Industrial printed matter image registration method and device based on lightweight network
CN117036427A
Moving object tracking device
JP2020098590A
Cited By
Method and system for monitoring operation behaviors of construction personnel of power distribution network
CN119810727A
