An instance segmentation-based power worker goggles wearing detection method

Abstract

The application discloses to the electric power technical field, specifically for a kind of electric power worker goggles wearing detection method based on instance segmentation, including specific steps as follows: the data set D1 of original electric power worker wearing goggles operation image is constructed;D1 is preprocessed, and the data set D2 of new electric power worker wearing goggles operation image is obtained;An EGSNet network is designed, and the network is used to process any electric power worker operation image in D2, and the wearing detection result of electric power worker goggles is obtained.The application generates noise filter related to each goggles instance dynamically through noise disturbance and reverse noise reduction mechanism, effectively captures the boundary features of goggles, realizes more accurate boundary processing;Mask feature map provides global context information, further enhances the robustness of segmentation in complex background, significantly reduces the false rejection rate and false detection rate, especially when the goggles and background texture or color are similar.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric power, in particular to a method for detecting wearing of electric power worker's goggles based on instance segmentation. BACKGROUND

[0002] In electric power operation, goggles are a key part of the protective equipment for workers, which can effectively protect the eyes from potential harm during operation. However, in actual operation environment, due to the reasons such as heavy goggles equipment, uncomfortable wearing, and personal use habits, electric power workers may not wear or even not wear goggles correctly, which may cause operation safety risks. Therefore, real-time monitoring of the wearing situation of electric power workers' goggles has become one of the important measures to ensure operation safety.

[0003] At present, common detection methods are mostly based on semantic segmentation or image detection technology. Although these methods can judge the wearing situation of goggles to some extent, they have obvious shortcomings in actual application. The semantic segmentation method identifies the target by segmenting the image into different category regions, but in the complex scene of electric power operation, especially in the case of complex background (such as environment with similar color to goggles), target being occluded or light condition changing, its ability to process fine-grained boundaries of goggles is insufficient, which may lead to blurred boundaries or missed targets; the image detection method mainly identifies the existence of the target, but cannot accurately judge the wearing state of the goggles (such as position, angle and tightness), so it is difficult to meet the high requirements of detection accuracy in actual operation.

[0004] Based on the above, the current detection of electric power worker's goggles wearing mainly adopts the method of deep learning, which has the following shortcomings:

[0005] 1. The existing method pays more attention to high-level semantic information, and ignores the rich local information in shallow features, resulting in insufficient robustness of the model to small targets;

[0006] 2. Most of the existing methods directly generate masks based on feature maps, but feature maps cannot accurately capture the fine-grained boundary information of instances, and cannot generate fine goggles boundaries, which may lead to blurred goggles boundaries when generating masks; the above problems will be particularly obvious in the case of complex target shape or occlusion;

[0007] 3. The existing method often has missed detection or false detection when dealing with complex background, especially when the texture or color between the background and the goggles is similar (such as the color of the goggles is similar to the surrounding environment), and when the goggles are not worn correctly (such as the goggles do not completely cover the goggles wearing area or are worn loosely), the misjudgment rate is high.

[0008] Therefore, the application provides a power worker goggles wearing detection method based on instance segmentation to solve the problems of insufficient complex background processing, boundary refinement, multi-instance distinguishing capability and target state sensing capability in the prior art; by combining multi-scale feature enhancement, goggle prompt embedded features and noise disturbance mechanisms, the robustness and accuracy of the goggle wearing detection are significantly improved, an efficient and reliable solution is provided for safety monitoring of power operation, the safety of power operation is greatly improved, the risk of injury is reduced, and more reliable operation guarantee is provided for power enterprises. SUMMARY

[0009] To solve the above technical problems, according to one aspect of the application, the application provides the following technical solutions:

[0010] A power worker goggles wearing detection method based on instance segmentation, comprising the following specific steps:

[0011] S1: constructing an original power worker wearing goggles operation image dataset D1;

[0012] S2: preprocessing D1 to obtain a new power worker wearing goggles operation image dataset D2;

[0013] S3: designing an EGSNet network, and using the network to process any power worker operation image in D2 to obtain a power worker goggles wearing detection result;

[0014] S4: loading a training set and a verification set in D2 into the EGSNet network for training to update network parameters, and taking a test set in D2 as input to verify network effect;

[0015] S5: applying the trained EGSNet network to a goggle detection task, and integrating the network into a power construction site monitoring system;

[0016] The specific steps of S3 are as follows:

[0017] S31: constructing a multi-scale feature map generation module to generate a multi-scale feature map enhanced in features;

[0018] S32: using a multi-scale feature map fusion module to fuse P1, P2, P3 and P4 of different scales, and outputting a mask feature map F mask ;

[0019] S33: constructing a Q2P-Cross module, and using a cross attention mechanism and a multi-layer perception machine MLP to perform feature fusion and enhancement processing on P1, P2, P3 and P4 to generate a goggle prompt embedded feature matrix;

[0020] S34: Construct a B2F-Diffu module to generate a noise filter NF related to each instance of the goggles;

[0021] S35: Use the BMD module to construct a power worker goggles instance mask feature map to obtain the power worker goggles wearing detection result.

[0022] As a preferred scheme of the power worker goggles wearing detection method based on instance segmentation, the specific steps of S1 are as follows:

[0023] S11: Install a control ball and a camera shooting device in a power grid construction site to collect power worker operation images under different light, weather, shooting angle and different posture conditions of the power workers;

[0024] S12: In all collected power worker operation images, the images of correctly wearing goggles are taken as positive samples;

[0025] S13: The images of not wearing, incorrectly wearing and wearing other glasses are taken as negative samples, and the number of positive and negative samples is relatively balanced, and finally the original power worker wearing goggles operation image dataset D1 is formed.

[0026] As a preferred scheme of the power worker goggles wearing detection method based on instance segmentation, the specific steps of S2 are as follows:

[0027] S21: Perform data cleaning operation on the collected D1 to remove blurred, low-quality and repeated samples;

[0028] S22: Construct ground truth label GT sample, which contains the boundary box, instance mask and instance identification annotation data of each instance in the image. Specifically, use the annotation tool to annotate D1, annotate the boundary box of the goggles wearing area of the power worker in D1, and the boundary box attribute contains its position information and category information, wherein the category information includes two kinds: wearing goggles, not wearing goggles; In addition, use the annotation tool to draw the instance mask of the power worker operation image wearing goggles in D1, and the power worker operation image not wearing goggles in D1 does not need to draw the instance mask;

[0029] S23: In order to enhance the diversity of data and improve the robustness of the model, data augmentation technology is adopted to perform random rotation, cropping, flipping and light adjustment operation on the images in D1 to obtain the data set after data augmentation;

[0030] S24: The data set after data enhancement is segmented, and is divided into a training set, a verification set and a test set according to a ratio of 8:1:1, so as to obtain a new data set D2 of power worker wearing goggles operation images.

[0031] As a preferred scheme of the power worker goggles wearing detection method based on instance segmentation, the specific steps of S31 are as follows:

[0032] S311: The image Y0 of the spatial resolution HxW and the channel number C in D2 is subjected to a pooling operation, so as to obtain a first goggle shallow feature map Y1 of the spatial resolution H / 2xW / 2 and the channel number C, and then subjected to twice Conv1x1 convolution operations, so as to obtain a second goggle shallow feature map Y2 of the spatial resolution H / 2xW / 2 and the channel number 2C;

[0033] S312: Y2 is input into a goggle wearing detection feature extraction sub-module for feature extraction, so as to obtain multi-scale feature maps, including a first layer feature map F1, a second layer feature map F2, a third layer feature map F3 and a fourth layer feature map F4;

[0034] S313: An F-Agg network is designed, a propagation mode from deep semantic features to shallow features is adopted to perform feature enhancement on F1, F2, F3 and F4, so as to obtain goggle feature enhanced multi-scale feature maps, including a first layer enhanced feature map P1, a second layer enhanced feature map P2, a third layer enhanced feature map P3 and a fourth layer enhanced feature map P4.

[0035] As a preferred scheme of the power worker goggles wearing detection method based on instance segmentation, the specific steps of S312 are as follows:

[0036] S3121: Y2 is input into a first layer Conv_Layer1 of a goggle wearing detection feature extraction sub-module, a Conv5x5 convolution operation is performed on Y2 by using a convolution kernel with a size of 5x5 to capture local features, a batch normalization BN operation is adopted to reduce the gradient vanishing problem, a ReLU activation function is used for operation to introduce nonlinearity and enhance the model expression ability, preliminary feature extraction is realized, and a first layer feature map F1 of goggle wearing detection is obtained.

[0037] S3122: input F1 to the second layer Conv_Layer2 of the goggle wearing detection feature extraction submodule, perform Conv1x1 convolution operation on F1 using a convolution kernel with a size of 1x1 to adjust the number of channels and change the depth of the feature map, perform BN operation and ReLU function operation to reduce gradient disappearance and introduce nonlinearity, then perform Conv3x3 convolution operation on the image using a convolution kernel with a size of 3x3 to capture local information in the image, perform Conv1x1 convolution operation on the feature map after recovering the number of channels using a convolution kernel with a size of 1x1, and perform BN operation and ReLU function operation on the feature map after recovering the number of channels to obtain the second layer feature map F2 of the goggle wearing detection;

[0038] S3123: after performing residual addition on F1 and F2, input to the third layer Conv_Layer3 of the goggle wearing detection feature extraction submodule, perform Conv1x1 convolution operation on F1 using a convolution kernel with a size of 1x1 to adjust the number of channels and change the depth of the feature map, perform BN operation and ReLU function operation to reduce gradient disappearance and introduce nonlinearity, then perform Conv3x3 convolution operation on the image using a convolution kernel with a size of 3x3 to capture local information in the image, perform Conv1x1 convolution operation on the feature map after recovering the number of channels using a convolution kernel with a size of 1x1, and perform BN operation and ReLU function operation on the feature map after recovering the number of channels to obtain the third layer feature map F3 of the goggle wearing detection;

[0039] S3124: after performing residual addition on F2 and F3, input to the fourth layer Conv_Layer4 of the goggle wearing detection feature extraction submodule, perform Conv1x1 convolution operation on F1 using a convolution kernel with a size of 1x1 to adjust the number of channels and change the depth of the feature map, perform BN operation and ReLU function operation to reduce gradient disappearance and introduce nonlinearity, then perform Conv3x3 convolution operation on the image using a convolution kernel with a size of 3x3 to capture local information in the image, perform Conv1x1 convolution operation on the feature map after recovering the number of channels using a convolution kernel with a size of 1x1, and perform BN operation and ReLU function operation on the feature map after recovering the number of channels to obtain the fourth layer feature map F4 of the goggle wearing detection;

[0040] The specific steps of S313 are as follows:

[0041] S3131: input F4 as input, in order to unify the number of channels, perform Conv1x1 convolution operation on F4 using a convolution kernel with a size of 1x1 to obtain a feature map F'4, perform Conv3x3 convolution operation on F'4 using a convolution kernel with a size of 3x3 to enhance spatial information, and perform ReLU activation function operation to introduce nonlinearity, to obtain the fourth layer enhanced feature map P4 of the goggle wearing detection;

[0042] S3132: Conv1x1 convolution operation is performed on F3 using a convolution kernel with a size of 1x1 to obtain a feature map F'3, and the feature maps F'4, F'3 and P4 are added to obtain M3; Conv3x3 convolution operation is performed on M3 using a convolution kernel with a size of 3x3 to obtain a third layer of enhanced feature maps P3 for the goggle wearing detection;

[0043] S3133: Conv1x1 convolution operation is performed on F2 using a convolution kernel with a size of 1x1 to obtain a feature map F'2, and the feature maps M3, F'2 and P3 are added to obtain M2; Conv3x3 convolution operation is performed on M2 using a convolution kernel with a size of 3x3 to obtain a second layer of enhanced feature maps P2 for the goggle wearing detection;

[0044] S3134: F1 is taken as input, and a feature map F'1 is obtained after Conv1x1 convolution, and the feature maps M2, F'1 and P2 are added to obtain M1; Conv3x3 operation is performed on M1 to obtain a first layer of enhanced feature maps P1 for the goggle wearing detection;

[0045] S3135: After convolution operation and fusion of feature maps, the multi-scale feature maps required for goggle wearing detection are enhanced, and finally the multi-scale feature maps P1, P2, P3 and P4 of goggle feature enhancement are obtained;

[0046] In the F-Agg network, the definition of goggle feature enhancement of F1, F2, F3 and F4 is as follows:

[0047] F' i = Φ Conv1×1 (F i ),1≤i≤4

[0048] P j = Φ Conv3×3 (M j ),1≤j≤4

[0049] M k = M k+1 + P k+1 + F' k ,1≤k≤3

[0050] Wherein, F i is the input multi-scale feature map; Φ Conv1×1 represents Conv1x1 convolution operation, which is used to adjust the channel number by convolution; F' iF'1 is the feature map after adjusting the channel, F'2 is the feature map after adjusting the channel, F'3 is the feature map after adjusting the channel, F'4 is the feature map after adjusting the channel, and Φ Conv3×3 Conv3x3 convolution operation is used to enhance spatial information, M j M4 is the deepest feature map, that is, F'4, M3 is the feature map obtained by adding M4, P4 and F'3 feature fusion, M2 is the feature map obtained by adding M3, P3 and F'2 feature fusion, and M1 is the feature map obtained by adding M2, P2 and F'1 feature fusion; P j M4 is the deepest feature map, that is, F'4, M3 is the feature map obtained by adding M4, P4 and F'3 feature fusion, M2 is the feature map obtained by adding M3, P3 and F'2 feature fusion, and M1 is the feature map obtained by adding M2, P2 and F'1 feature fusion; P

[0051] As a preferred scheme of the power worker goggles wearing detection method based on instance segmentation, the specific steps of S32 are as follows:

[0052] S321: Fusion operation is performed on P1, P2, P3 and P4 to obtain global semantic information and fine local detail information, and the fifth layer enhanced feature map P5 after fusion is obtained;

[0053] S322: Conv1x1 convolution operation is performed on P5 using a convolution kernel with a size of 1x1, and then batch normalization BN operation and ReLU function operation are performed, to obtain the mask feature map F mask .

[0054] As a preferred scheme of the power worker goggles wearing detection method based on instance segmentation, the specific steps of S33 are as follows:

[0055] S331: Cross attention mechanism is used to perform attention weight operation on P1, P2, P3 and P4 to obtain goggles query matrix, including first goggles query matrix Q1, second goggles query matrix Q2, third goggles query matrix Q3 and fourth goggles query matrix Q4;

[0056] S332: Feature enhancement is performed on Q1, Q2, Q3 and Q4 by using a multi-layer perception MLP to generate a goggle prompt embedding matrix, including a first goggle prompt embedding matrix E1, a second goggle prompt embedding matrix E2, a third goggle prompt embedding matrix E3 and a fourth goggle prompt embedding matrix E4, which are used to guide the subsequent process of mask generation, and the formula is:

[0057] E i = Φ mlp (Q i ), 1≤i≤4

[0058] Wherein, Q i represents a goggle query matrix; Φ mlp represents a multi-layer perception MLP, which is composed of multiple fully connected layers and ReLU activation functions; E i represents a goggle prompt embedding matrix, wherein E1 is a first goggle prompt embedding matrix obtained by performing multi-layer perception operation on Q1, E2 is a second goggle prompt embedding matrix obtained by performing multi-layer perception operation on Q2, E3 is a third goggle prompt embedding matrix obtained by performing multi-layer perception operation on Q3, and E4 is a fourth goggle prompt embedding matrix obtained by performing multi-layer perception operation on Q4;

[0059] S333: The E1, E2, E3 and E4 are transformed by a sine function sin to generate a goggle prompt embedding feature matrix, including a first goggle prompt embedding feature matrix T1, a second goggle prompt embedding feature matrix T2, a third goggle prompt embedding feature matrix T3, and a fourth goggle prompt embedding feature matrix T4, which are used to provide position information and shape feature prompts of the goggle, and the calculation formula is:

[0060] T i =E i +sin(E i ), 1≤i≤4

[0061] Wherein, E i represents a goggle prompt embedding matrix; sin(E i ) represents a sine transformation of E i ; T i represents a goggle prompt embedding feature matrix, wherein T1 is a first goggle prompt embedding feature matrix obtained by adding E1 and sin(E1), T2 is a second goggle prompt embedding feature matrix obtained by adding E2 and sin(E2), T3 is a third goggle prompt embedding feature matrix obtained by adding E3 and sin(E3), and T4 is a fourth goggle prompt embedding feature matrix obtained by adding E4 and sin(E4);

[0062] The specific steps of S331 are as follows:

[0063] S3311: preset Q0 is a goggle query matrix initialized as zero, the dimension of Q0 is (N q ,d), wherein N q represents the number of query features, d represents the feature dimension of the query, the cross attention mechanism is used to calculate the attention weight between P1 and Q0, the attention weight matrix A1 is obtained by Softmax normalization, A1 is added to Q0 to generate the first goggle query matrix Q1;

[0064] S3312: the cross attention mechanism is used to calculate the attention weight between P2 and Q1, the attention weight matrix A2 is obtained by Softmax normalization, A2 is added to Q1 to generate the second goggle query matrix Q2;

[0065] S3313: the cross attention mechanism is used to calculate the attention weight between P3 and Q2, the attention weight matrix A3 is obtained by Softmax normalization, A3 is added to Q2 to generate the third goggle query matrix Q3;

[0066] S3314: the cross attention mechanism is used to calculate the attention weight between P4 and Q3, the attention weight matrix A4 is obtained by Softmax normalization, A4 is added to Q3 to generate the fourth goggle query matrix Q4, and the formula is:

[0067]

[0068] Q i =A i +Q i-1 ,1≤i≤4

[0069] Wherein, A i represents the attention weight matrix calculated by the cross attention mechanism; Q i represents the goggle query matrix; the dimensions of A i and Q i are both (N q ,d), N q represents the number of query features; d represents the feature dimension of the query; P i is a feature map, and the dimension is (HxW, d); Φ attention (P i ,Q i-1 ) represents a cross attention function, which is used to calculate the information interaction between the goggle query matrix Q i-1 and the feature map P i ; W Q , W K and W V are linear transformation matrices, which are added to Qi-1 and P i projected to the same feature dimension d; is N q is a matrix of H rows and W columns, representing attention weights; Softmax represents an activation function, ensuring that the weights are between 0 and 1 and the sum is 1, normalizing the attention weight distribution; by A i and Q i-1 are added to obtain Q i , which can accumulate the information obtained from the feature map P i , so that it gradually learns the key features of the goggles.

[0070] As a preferred scheme of the power worker goggles wearing detection method based on instance segmentation, the specific steps of S34 are as follows:

[0071] S341: Perturb GT, add Gaussian noise to GT samples step by step in the forward Forword stage to generate NBB, and the specific formula of the noise perturbation process is as follows:

[0072]

[0073] where q(x t |x t-1 ) represents the noise perturbation process, x0 represents the real image without noise, t represents the t-th time of adding Gaussian noise, T represents the total number of adding noise, represents a Gaussian distribution, β t represents the variance of the Gaussian distribution, which is used to control the size of the added noise, I represents the covariance matrix of the noise, which is a unit matrix, and the size is consistent with the dimension of the input data x t or x t-1 ;

[0074] S342: Sample a Gaussian independent noise variable ∈ ~ N(0, I) from x0 to obtain the sample of x t , and the specific formula is as follows:

[0075]

[0076] where, is a weight coefficient, representing the cumulative value of the variance at each time step; ∈ represents a Gaussian independent noise variable, which is random noise introduced in each step of the diffusion process, and as t increases, x t is closer and closer to pure noise, and when T→∞, x T is completely Gaussian noise, and the goggles noise bounding box NBB ∈ {x1, x2, …, x T} is obtained.

[0077] S343: input NBB, T1, T2, T3 and T4 into the reverse stage for noise reduction processing, which is used to reverse the noise adding process and sample from p(x t-1 |x t ), the reverse process formula is as follows:

[0078]

[0079] In the noise reduction process, represents a Gaussian distribution; μ(x t ,t) represents the mean, ∑(x t ,t) is the covariance, and the mean and covariance are optimized and learned by minimizing the loss function and parameterized by a neural network; p(x t-1 |x t ) represents the state x t at the previous moment according to the current state x t-1 , the model recovers the image features before noise adding through learning parameters, and in this process, the boundary box feature BBF after removing noise is obtained;

[0080] S344: send BBF into the full connection layer FC, and after linear transformation and ReLU activation function operation, generate the noise filter NF related to the goggle instance, and the process of generating NF is as follows:

[0081] NF=η(f(x t ,t))

[0082] Wherein, η represents the full connection layer, which is used to learn the mapping relationship between the noise-removed features and the filter, and f(x t ,t) is the boundary box feature after removing noise.

[0083] As a preferred scheme of the power worker goggle wearing detection method based on instance segmentation, wherein: the specific steps of S35 are as follows:

[0084] S351: use NF to perform convolution operation on F mask , extract the shape, boundary and other key features of the specific goggle instance, and obtain the feature map S1 with boundary box information, S1 contains the position and size information of the boundary box, then use the Conv1x1 convolution operation with the size of 1x1 to adjust the channel number of the feature map, thereby obtaining S2.

[0085] S352: using a sigmoid activation function operation on S2 to constrain the output value of S2 in [0, 1], specifically: setting a pixel judgment threshold H1, binarizing the output value of S2, if the output value is less than H1, it means that the pixel belongs to the background, and the corresponding output value is 0; if the output value is greater than H1, it means that the pixel belongs to the goggle instance, and the corresponding output value is 1, and finally an electric power worker goggle instance mask feature map BM composed of 0 and 1 is obtained, wherein 0 represents that the pixel point belongs to the background area, and 1 represents that the pixel point belongs to the goggle area;

[0086] S353: setting a goggle wearing detection threshold H2, in the boundary box area of the goggle instance, if the output goggle area in BM is greater than H2, it is judged that the goggle has been worn, if the output goggle area in BM is less than H2, it is judged that the goggle has not been worn, and the wearing detection result of the electric power worker goggle can be obtained.

[0087] As a preferred scheme of the electric power worker goggle wearing detection method based on instance segmentation, the specific steps of S4 are as follows:

[0088] S41: according to D2, training and verifying the EGSNet network constructed for S3, first initializing all neural network parameters and related hyperparameters;

[0089] S42: after initializing the parameters, dividing the training set in D2 into multiple batches according to the batch size, inputting the batches for training to obtain the training loss value loss of each batch, and after all batches of the training set are trained, dividing the verification set in D2 according to the batch size and inputting it to obtain the corresponding batch loss batch_loss;

[0090] S43: in the training and verification, the algorithm learns and adjusts the parameters according to the corresponding loss value, and the training process is trained for multiple rounds according to the preset training rounds, and when the algorithm training reaches the loss value convergence, the instance segmentation algorithm training of the electric power worker goggle wearing detection is completed;

[0091] S44: after the training and verification are completed, the test set preprocessed by D2 is applied, the test data is input into the instance algorithm of the electric power worker goggle wearing detection with the optimal network parameters after training, the detection test of the electric power worker goggle wearing situation is carried out, the accuracy and the recall rate are used as the verification indexes of the algorithm, and the network effect is verified;

[0092] The specific steps of S5 are as follows:

[0093] S51: The trained EGSNet network is used as one of the modules of the power construction site monitoring system, which is connected with the camera of the power construction site to monitor the wearing of the power worker's goggles in real time;

[0094] S52: If the wearing is not standardized, the system will alarm, and the relevant image will be recorded in the system for relevant personnel to take corresponding processing measures.

[0095] Compared with the prior art:

[0096] 1. The present application generates a noise filter related to each instance of goggles through noise disturbance and reverse noise reduction mechanism, effectively captures the boundary features of the goggles, and realizes more accurate boundary processing; the mask feature map provides global context information, further enhances the robustness of segmentation in complex background, significantly reduces the false detection rate and false detection rate, especially when the goggles and background texture or color are similar;

[0097] 2. The present application combines the embedded feature matrix of the goggles prompt to dynamically adapt to the positioning and shape change of the goggles, generates features for each instance, can accurately distinguish multiple target instances, and avoids mask overlap; at the same time, the multi-scale feature fusion of the goggles retains the shallow detail information, improves the perception and segmentation ability of small targets, and adapts to the segmentation demand of diversified scenes. BRIEF DESCRIPTION OF DRAWINGS

[0098] Figure 1 The flowchart of the present application is shown;

[0099] Figure 2 The EGSNet network structure diagram of the present application is shown;

[0100] Figure 3 The multi-scale feature map generation module diagram of the present application is shown;

[0101] Figure 4 The goggle wearing detection feature extraction sub-module diagram of the present application is shown;

[0102] Figure 5 The F-Agg network structure diagram of the present application is shown;

[0103] Figure 6 The multi-scale feature fusion module diagram of the present application is shown;

[0104] Figure 7 The Q2P-Cross module diagram of the present application is shown;

[0105] Figure 8 The B2F-Diffu module diagram of the present application is shown;

[0106] Figure 9The schematic diagram of the BMD module of the present application. DETAILED DESCRIPTION

[0107] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0108] The present application provides a power worker goggles wearing detection method based on instance segmentation, please refer to Figures 1-9 , including the specific steps as follows:

[0109] S1: constructing a data set D1 of original power worker wearing goggles operation images;

[0110] The specific steps of S1 are as follows:

[0111] S11: installing a control ball and a camera shooting device including but not limited to in a power grid construction operation site, collecting power worker operation images under different light, weather, shooting angles and different postures of power workers;

[0112] S12: in all collected power worker operation images, taking the images of correctly wearing goggles as positive samples;

[0113] S13: taking the images of not wearing, incorrectly wearing and wearing other glasses as negative samples, the number of positive and negative samples is relatively balanced, and finally forming a data set D1 of original power worker wearing goggles operation images;

[0114] S2: preprocessing D1 to obtain a new data set D2 of power worker wearing goggles operation images;

[0115] The specific steps of S2 are as follows:

[0116] S21: performing data cleaning operation on collected D1, and eliminating blurred, low-quality and repeated samples;

[0117] S22: constructing a ground truth GT sample, the GT sample contains the boundary box, instance mask and instance identification annotation data of each instance in the image, specifically, using an annotation tool to annotate D1, annotating the boundary box of the goggles wearing area of the power worker in D1, the boundary box attribute contains its position information and category information, wherein the category information includes two kinds: wearing goggles, not wearing goggles; in addition, using an annotation tool to draw an instance mask for the power worker operation image wearing goggles in D1, the power worker operation image not wearing goggles in D1 does not need to draw an instance mask;

[0118] S23: In order to enhance the diversity of data and improve the robustness of the model, data augmentation technology will be used to randomly rotate, crop, flip and adjust the light of the images in D1 to obtain the data set after data augmentation;

[0119] S24: The data set after data augmentation is segmented, and the training set, the validation set and the test set are divided according to the ratio of 8:1:1 to obtain a new data set D2 of power workers wearing goggles in operation images;

[0120] In the experiment of the application, the existing LabelMe software is used as the labeling tool, and the software is not implemented in the patent; data cleaning: manually remove blurred, low-quality and repeated image samples to ensure the quality of the data set; instance mask: a pixel-level binary labeling matrix (0 / 1) used to accurately identify the contour area of each goggle wearing instance in the image; random rotation: taking the center of the image as the origin, randomly generating a rotation angle of-15° to +15°, and filling the blank area by bilinear interpolation; cropping: randomly generating a cropping ratio in the range of 80%-95% of the image to retain the goggle wearing area of the power worker in the image; flipping: horizontally flipping the image; light adjustment: adjusting the RGB three-channel values by linear transformation of R±20, G±15 and B±10 to finally generate an enhanced image;

[0121] S3: Design an EGSNet network, and use the network to process any power worker operation image in D2 to obtain the detection result of the power worker goggles;

[0122] An electric power worker goggles segmentation network (EGSNet) is designed to detect whether the power worker wears goggles from the power worker operation image; the network includes the following modules: a multi-scale feature map generation module, a multi-scale feature map fusion module, a Q2P-Cross module, a B2F-Diffu module and a BMD module; the structure of the EGSNet network is as shown in Figure 2 ;

[0123] The specific steps of S3 are as follows:

[0124] S31: Construct a multi-scale feature map generation module to generate a multi-scale feature map enhanced by features; the structure of the multi-scale feature map generation module is as shown in Figure 3 ;

[0125] The specific steps of S31 are as follows:

[0126] S311: Perform a pooling operation on the image Y0 with a spatial resolution of HxW and a channel number of C to obtain a first goggle shallow feature map Y1 with a spatial resolution of H / 2xW / 2 and a channel number of C, and then perform twice Conv1x1 convolution operations to obtain a second goggle shallow feature map Y2 with a spatial resolution of H / 2xW / 2 and a channel number of 2C;

[0127] S312: Input Y2 into a goggle wearing detection feature extraction sub-module to perform feature extraction to obtain a multi-scale feature map including a first layer feature map F1, a second layer feature map F2, a third layer feature map F3, and a fourth layer feature map F4; the specific structure of the goggle wearing detection feature extraction sub-module is as shown in Figure 4

[0128] The specific steps of S312 are as follows:

[0129] S3121: Input Y2 into the first layer Conv_Layer1 of the goggle wearing detection feature extraction sub-module, perform Conv5x5 convolution operation on Y2 using a convolution kernel with a size of 5x5 to capture local features, perform Batch Normalization (BN) operation to reduce the gradient vanishing problem, perform ReLU activation function operation to introduce nonlinearity and enhance the model expression ability, and realize preliminary feature extraction to obtain the first layer feature map F1 of the goggle wearing detection;

[0130] S3122: Input F1 into the second layer Conv_Layer2 of the goggle wearing detection feature extraction sub-module, perform Conv1x1 convolution operation on F1 using a convolution kernel with a size of 1x1 to adjust the channel number and change the depth of the feature map, perform BN operation and ReLU function operation to reduce the gradient vanishing and introduce nonlinearity, perform Conv3x3 convolution operation on the feature map using a convolution kernel with a size of 3x3 to capture local information in the image, perform Conv1x1 convolution operation on the feature map using a convolution kernel with a size of 1x1 to restore the channel number, and perform BN operation and ReLU function operation on the feature map after the channel number is restored to obtain the second layer feature map F2 of the goggle wearing detection;

[0131] ​S3123: After residual addition of F1 and F2, input to the third layer Conv_Layer3 of the goggle wearing detection feature extraction submodule, perform Conv1x1 convolution operation with a convolution kernel of 1x1 to adjust the number of channels and change the depth of the feature map, perform BN operation and ReLU function operation to reduce gradient disappearance and introduce nonlinearity, then perform Conv3x3 convolution operation with a convolution kernel of 3x3 to capture local information in the image, perform Conv1x1 convolution operation with a convolution kernel of 1x1 to restore the number of channels, and perform BN operation and ReLU function operation on the feature map after restoring the channels to obtain the third layer feature map F3 of the goggle wearing detection;

[0132] S3124: After residual addition of F2 and F3, input to the fourth layer Conv_Layer4 of the goggle wearing detection feature extraction submodule, perform Conv1x1 convolution operation with a convolution kernel of 1x1 to adjust the number of channels and change the depth of the feature map, perform BN operation and ReLU function operation to reduce gradient disappearance and introduce nonlinearity, then perform Conv3x3 convolution operation with a convolution kernel of 3x3 to capture local information in the image, perform Conv1x1 convolution operation with a convolution kernel of 1x1 to restore the number of channels, and perform BN operation and ReLU function operation on the feature map after restoring the channels to obtain the fourth layer feature map F4 of the goggle wearing detection;

[0133] The goggle wearing detection feature extraction submodule has the advantages that as the number of layers increases, the number of channels and the depth also increase, and more and more complex and abstract features can be extracted, so that the scales of the final feature maps F1, F2, F3 and F4 are complementary and the same, thereby forming a series of multi-scale feature maps;

[0134] S313: Design an F-Agg network to enhance the features of F1, F2, F3 and F4 in a propagation manner from deep semantic features to shallow features, and obtain goggle feature-enhanced multi-scale feature maps, including a first layer enhanced feature map P1, a second layer enhanced feature map P2, a third layer enhanced feature map P3 and a fourth layer enhanced feature map P4;

[0135] The specific steps of S313 are as follows:

[0136] S3131: Take F4 as input, perform Conv1x1 convolution operation on F4 with a convolution kernel of 1x1 to F'4 to unify the number of channels, perform Conv3x3 convolution operation on F'4 with a convolution kernel of 3x3 to enhance spatial information, perform ReLU activation function operation to introduce nonlinearity, and obtain the fourth layer enhanced feature map P4 of the goggle wearing detection;

[0137] S3132: Conv1x1 convolution operation is performed on F3 using a convolution kernel with a size of 1x1 to obtain a feature map F'3, and the feature maps F'4, F'3 and P4 are added to obtain M3; Conv3x3 convolution operation is performed on M3 using a convolution kernel with a size of 3x3 to obtain a third layer of enhanced feature maps P3 for the goggle wearing detection;

[0138] S3133: Conv1x1 convolution operation is performed on F2 using a convolution kernel with a size of 1x1 to obtain a feature map F'2, and the feature maps M3, F'2 and P3 are added to obtain M2; Conv3x3 convolution operation is performed on M2 using a convolution kernel with a size of 3x3 to obtain a second layer of enhanced feature maps P2 for the goggle wearing detection;

[0139] S3134: F1 is taken as input, and a feature map F'1 is obtained after Conv1x1 convolution, and the feature maps M2, F'1 and P2 are added to obtain M1; Conv3x3 operation is performed on M1 to obtain a first layer of enhanced feature maps P1 for the goggle wearing detection;

[0140] S3135: After convolution operation and fusion of feature maps, the multi-scale feature maps required for goggle wearing detection are enhanced, and finally the multi-scale feature maps P1, P2, P3 and P4 of goggle feature enhancement are obtained;

[0141] In the F-Agg network, the definition of goggle feature enhancement of F1, F2, F3 and F4 is as follows:

[0142] F' i = Φ Conv1×1 (F i ),1≤i≤4

[0143] P j = Φ Conv3×3 (M j ),1≤j≤4

[0144] M k = M k+1 + P k+1 + F' k ,1≤k≤3

[0145] Where, F i is the input multi-scale feature map; Φ Conv1×1 represents Conv1x1 convolution operation, which is used to adjust the channel number by convolution; F' iRepresents the feature map after channel adjustment, where F'1 is the feature map after channel adjustment obtained by performing Conv1×1 convolution operation on F1, F'2 is the feature map after channel adjustment obtained by performing Conv1×1 convolution operation on F2, F'3 is the feature map after channel adjustment obtained by performing Conv1×1 convolution operation on F3, and F'4 is the feature map after channel adjustment obtained by performing Conv1×1 convolution operation on F4; Φ Conv3×3 Conv3×3 convolution operation is used to enhance spatial information, M j Represents the feature map after the goggles feature fusion, where M4 is the deepest feature map, namely F'4, M3 is the feature map obtained by fusing and adding the features of M4, P4 and F'3, M2 is the feature map obtained by fusing and adding the features of M3, P3 and F'2, and M1 is the feature map obtained by fusing and adding the features of M2, P2 and F'1; P j It is a multi-scale feature map of goggles feature enhancement, P1 is the first layer enhanced feature map obtained by performing Conv3×3 convolution operation on M1, P2 is the second layer enhanced feature map obtained by performing Conv3×3 convolution operation on M2, P3 is the third layer enhanced feature map obtained by performing Conv3×3 convolution operation on M3, and P4 is the fourth layer enhanced feature map obtained by performing Conv3×3 convolution operation on M4;

[0146] S32: Use the multi-scale feature map fusion module to fuse P1, P2, P3 and P4 of different scales and output a mask feature map F mask ; The structure of the multi-scale feature map fusion module is as follows Figure 6 As shown;

[0147] The specific steps of S32 are as follows:

[0148] S321: Perform a fusion operation on P1, P2, P3 and P4 to obtain global semantic information and fine local detail information, and obtain the fused fifth-layer enhanced feature map P5;

[0149] Among them, the Fusion operation: first, the multi-scale feature maps P1, P2, P3 and P4 are bilinearly interpolated and upsampled to a unified resolution, then the layer weight coefficients are calculated through the channel attention mechanism, and finally the weighted sum is performed to generate the fifth layer enhanced feature map P5;

[0150] S322: After performing Conv1×1 convolution operation on P5 using a convolution kernel of size 1×1, batch normalization BN operation and ReLU function operation are performed to obtain the mask feature map F mask ;

[0151] S33: Constructing the Q2P-Cross module, using cross-attention mechanism and multi-layer perception (MLP) to perform feature fusion and enhancement processing on P1, P2, P3 and P4, and generating a protective eyewear prompt embedding feature matrix;

[0152] The specific steps of S33 are as follows:

[0153] S331: Using cross-attention mechanism to perform attention weight operation on P1, P2, P3 and P4 to obtain a protective eyewear query matrix, including a first protective eyewear query matrix Q1, a second protective eyewear query matrix Q2, a third protective eyewear query matrix Q3 and a fourth protective eyewear query matrix Q4;

[0154] The specific steps of S331 are as follows:

[0155] S3311: Presetting Q0 as a protective eyewear query matrix initialized to zero, the dimension of Q0 is (N q ,d), where N q represents the number of features of the query, d represents the feature dimension of the query, the cross-attention mechanism is used to calculate the attention weight between P1 and Q0, the Softmax normalization is used to obtain the attention weight matrix A1, A1 is added to Q0 to generate the first protective eyewear query matrix Q1;

[0156] S3312: Using cross-attention mechanism to calculate the attention weight between P2 and Q1, using Softmax normalization to obtain the attention weight matrix A2, and adding A2 and Q1 to generate the second protective eyewear query matrix Q2;

[0157] S3313: Using cross-attention mechanism to calculate the attention weight between P3 and Q2, using Softmax normalization to obtain the attention weight matrix A3, and adding A3 and Q2 to generate the third protective eyewear query matrix Q3;

[0158] S3314: Using cross-attention mechanism to calculate the attention weight between P4 and Q3, using Softmax normalization to obtain the attention weight matrix A4, and adding A4 and Q3 to generate the fourth protective eyewear query matrix Q4, the formula is:

[0159]

[0160] Q i =A i +Q i-1 ,1≤i≤4

[0161] Wherein, A i represents the attention weight matrix calculated by cross-attention mechanism; Q i represents the protective eyewear query matrix; A i and Qi The dimensions are (N q ,d),N q represents the number of query features; d represents the feature dimension of the query; P i is a feature map with a dimension of (H×W,d); Φ attention (P i ,Q i-1 ) represents the cross attention function, which is used to calculate the goggles query matrix Q i-1 and feature map P i Information interaction between Q 、W K and W V is a linear transformation matrix, Q i-1 and P i Project to the same feature dimension d; Yes N q A matrix of H×W rows and columns represents the attention weights; Softmax represents the activation function, which ensures that the weights are between 0 and 1 and the sum is 1, normalizing the distribution of attention weights; i and Q i-1 Add up to get Q i , can accumulate from the feature map P i The information obtained allows it to gradually learn the key features of the goggles;

[0162] The goggles query matrix is ​​a set of learnable vectors used to detect and identify goggles. Initially set to zero, it extracts information from the feature map through a cross-attention mechanism and is gradually updated to enhance target perception. In each iteration, the query matrix calculates attention weights with the feature map, focuses on the goggles area, and updates itself using weighted feature information, gradually learning the position and shape characteristics of the goggles. Ultimately, after multiple rounds of updates, the goggles query matrix can accurately represent the target.

[0163] Cross attention mechanism: It is a method to calculate the information interaction between two different feature spaces (such as query matrix and feature map); it calculates the goggles query matrix Q i and feature map P i The similarity between them is normalized using Softmax to obtain the attention weight matrix A i , and then use the weight to perform weighted summation on the feature map, thereby injecting the key information of the feature map into the query matrix, realizing feature fusion, and enabling the query to focus on the important areas of the feature map, thereby improving the recognition ability and expression effect of the target;

[0164] Cross-attention weight calculation: Calculate the goggles query matrix Q i-1 and feature map P ito extract the key features of the goggles; the specific calculation process is as follows: first, the query matrix Q i-1 and the feature map P i are projected to the same feature dimension d through the linear transformation matrix W Q and W K ; then, the dot product of Q i-1 and P i after projection is calculated and divided by for scaling to prevent the gradient from being too large; finally, the similarity score is normalized using Softmax to convert it into a probability distribution, multiplied by P i W V to aggregate the features to obtain the attention weight matrix A i .

[0165] S332: use the multi-layer perception MLP to perform feature enhancement on Q1, Q2, Q3 and Q4 to generate the goggles prompt embedding matrix, including the first goggles prompt embedding matrix E1, the second goggles prompt embedding matrix E2, the third goggles prompt embedding matrix E3 and the fourth goggles prompt embedding matrix E4, which are used to guide the subsequent mask generation process, and the formula is:

[0166] E i =Φ mlp (Q i ),1≤i≤4

[0167] where Q i represents the goggles query matrix; Φ mlp represents the multi-layer perception MLP, which consists of multiple fully connected layers and ReLU activation functions; E i represents the goggles prompt embedding matrix, where E1 is the first goggles prompt embedding matrix obtained by performing multi-layer perception operation on Q1, E2 is the second goggles prompt embedding matrix obtained by performing multi-layer perception operation on Q2, E3 is the third goggles prompt embedding matrix obtained by performing multi-layer perception operation on Q3, and E4 is the fourth goggles prompt embedding matrix obtained by performing multi-layer perception operation on Q4;

[0168] S333: generate the goggles prompt embedding feature matrix by sine transformation sin of E1, E2, E3 and E4, including the first goggles prompt embedding feature matrix T1, the second goggles prompt embedding feature matrix T2, the third goggles prompt embedding feature matrix T3 and the fourth goggles prompt embedding feature matrix T4, and the above goggles prompt embedding feature matrix is used to provide the position information and shape feature prompt of the goggles, and the calculation formula is:

[0169] T i =E i +sin(Ei ),1≤i≤4

[0170] wherein, E i represents the eyewear cue embedding matrix; sin(E i ) represents the sine transformation of E i ; T i represents the eyewear cue embedding feature matrix, wherein T1 is the first eyewear cue embedding feature matrix obtained by adding E1 and sin(E1), T2 is the second eyewear cue embedding feature matrix obtained by adding E2 and sin(E2), T3 is the third eyewear cue embedding feature matrix obtained by adding E3 and sin(E3), and T4 is the fourth eyewear cue embedding feature matrix obtained by adding E4 and sin(E4);

[0171] S34: constructing a B2F-Diffu module to generate a noise filter NF related to each eyewear instance;

[0172] In the forward Forword stage, Gaussian noise is added to the GT sample to obtain an eyewear noise bounding box NBB (Noise Bounding Box), and then in the reverse Reverse stage, the NBB is denoised to obtain a bounding box feature BBF (Bounding Box Feature) after removing the noise, and then passes through the full connection layer FC to finally obtain the noise filter NF (Noisy Filter) related to the eyewear instance; the structure of the B2F-Diffu module is as shown in Figure 8

[0173] The specific steps of S34 are as follows:

[0174] S341: perturbing the GT, and entering the forward Forword stage to gradually add Gaussian noise to the GT sample to generate NBB, and the specific formula of the noise perturbation process is as follows:

[0175]

[0176] wherein, q(x t |x t-1 ) represents the noise perturbation process, x0 represents the real image without noise, t represents the tth time of adding Gaussian noise, T represents the total number of adding noise, represents the Gaussian distribution, β t represents the variance of the Gaussian distribution, which is used to control the size of the added noise, I represents the covariance matrix of the noise, which is a unit matrix, and the size is consistent with the dimension of the input data x t or x t-1 ;

[0177] ​S342: Sample a Gaussian independent noise variable ∈ ~ N(0, I) to x0 to obtain x t , the specific formula is as follows:

[0178]

[0179] Wherein, is a weight coefficient, representing the cumulative value of the variance of each time step; ∈ represents a Gaussian independent noise variable, which is a random noise introduced in each diffusion process, and as t increases, x t is closer and closer to pure noise, and when T→∞, x T is completely Gaussian noise, and the noise boundary box NBB of the goggles is obtained ∈{x1, x2,…, x T};

[0180] S343: Input NBB together with T1, T2, T3, T4 to the reverse stage (corresponding to Reverse in Figure 8 ) for noise reduction processing, which is used to reverse the noise adding process and sample from p(x t-1 |x t ), the formula of the reverse process is as follows:

[0181]

[0182] In the noise reduction process, represents a Gaussian distribution; μ(x t , t) represents the mean, and Σ(x t , t) is the covariance, and the mean and covariance are optimized and learned by minimizing the loss function, and are parameterized by a neural network; p(x t-1 |x t ) represents the state x t at the previous moment according to the current state x t-1 , and the model recovers the image features before noise adding through learning parameters, and in this process, the boundary box feature BBF after removing noise is obtained;

[0183] S344: Send BBF into the fully connected layer FC, and after linear transformation and ReLU activation function operation, generate the noise filter NF related to the goggle instance, and the process of generating NF is as follows:

[0184] NF=η(f(x t , t))

[0185] Wherein, η represents the fully connected layer, which is used to learn the mapping relationship between the noise-removed features and the filter, and f(x t , t) is the noise-removed boundary box feature;

[0186] S35: constructing the power worker goggles instance mask feature map by using the BMD module to obtain the power worker goggles wearing detection result;

[0187] using the NF on the F mask performing a series of convolution operations, and then using the ψ function on the F mask performing a series of transformations to generate the goggles instance mask feature map BM (Binary Mask), as follows:

[0188] BM = ψ (F mask ; NF) = MaxPool (BN (ReLU (F mask * NF)))

[0189] Wherein, the ψ function includes the ReLU activation function, the batch normalization BN operation and the maximum pooling MaxPool operation. After the transformation by the ψ function, the obtained BM can effectively represent the position and shape of each goggles instance in the image. The module is as shown in Figure 9 ;

[0190] The specific steps of the S35 are as follows:

[0191] S351: performing convolution operation on the F mask convolution by using the NF to extract the shape, boundary and other key features of the specific goggles instance, to obtain the feature map S1 with the boundary box information, S1 includes the position and size information of the boundary box, and then the size of the Conv1x1 convolution operation is 1x1, the channel number of the adjusted feature map is adjusted by performing convolution operation on S1, to obtain S2;

[0192] S352: using the Sigmoid activation function operation on S2 to constrain the output value of S2 between [0, 1], specifically: setting a pixel judgment threshold H1, binarizing the output value of S2, if the output value is less than H1, it means that the pixel belongs to the background, and the corresponding output value is 0; if the output value is greater than H1, it means that the pixel belongs to the goggles instance, and the corresponding output value is 1, finally obtaining a power worker goggles instance mask feature map BM composed of 0 and 1, where 0 represents that the pixel point belongs to the background area, and 1 represents that the pixel point belongs to the goggles area;

[0193] S353: setting a goggles wearing detection threshold H2, in the boundary box area of the goggles instance, if the output goggles area (i.e. the area of the pixel point being 1 in BM) in BM is greater than H2, it is judged as having worn the goggles, if the output goggles area (i.e. the area of the pixel point being 1 in BM) in BM is less than H2, it is judged as not having worn the goggles, and the power worker goggles wearing detection result can be obtained;

[0194] S4: Load the training set and the validation set in D2 into the EGSNet network for training to update the network parameters, and input the test set in D2 as an input to verify the network effect;

[0195] The specific steps of S4 are as follows:

[0196] S41: According to D2 (D2 includes a training set, a validation set and a test set), the EGSNet network constructed in S3 is trained and verified, and all neural network parameters and related hyperparameters are first initialized; for example, the training rounds, the batch size, the learning rate, the activation function, the loss function and the like;

[0197] S42: After the initialization of the parameters, the training set in D2 is divided into multiple batches according to the batch size, and the training is performed according to the batches to obtain the training loss value loss of each batch, and after the training of all batches of the training set is completed, the validation set in D2 is also divided according to the batch size and inputted to obtain the corresponding batch loss value batch_loss;

[0198] S43: In the training and verification, the algorithm learns and adjusts the parameters according to the corresponding loss value, and the training process is trained for multiple rounds according to the preset training rounds, and when the algorithm training converges to the loss value, the instance segmentation algorithm for power worker goggles wearing detection is trained;

[0199] S44: After the training and verification are completed, the test set preprocessed by D2 is applied, the test data is inputted to the instance algorithm for power worker goggles wearing detection with the optimal network parameters after the training, the detection test of the power worker goggles wearing situation is performed, and the accuracy and the recall rate are used as the verification indicators of the algorithm to verify the network effect;

[0200] S5: The trained EGSNet network is applied to the goggles detection task and integrated into the power construction site monitoring system;

[0201] The specific steps of S5 are as follows:

[0202] S51: The trained EGSNet network is used as one of the modules of the power construction site monitoring system, the system is connected with the camera of the power construction site, and the wearing situation of the power worker goggles is monitored in real time;

[0203] S52: If the wearing situation is not standard or not worn, the system will alarm, and the relevant image will be recorded in the system so that the relevant personnel can take corresponding processing measures.

[0204] The present application includes but is not limited to the following embodiments:

[0205] In the power operating environment, an industrial camera is arranged to collect operating image data of a power worker wearing goggles; after obtaining 1200 operating images of the power worker wearing goggles, a LabelMe labeling tool is used to label the images, and the labeling content includes a boundary box of a goggle wearing area and a wearing state (wearing goggles or not wearing goggles); after the labeling is completed, data preprocessing technology is used to enhance the images, including image rotation and cropping; during rotation, image rotation processing is performed using an OpenCV library, and cropping is performed by slicing the image using a Numpy library to ensure data set diversity, and finally 3600 images are obtained; then, the data set is divided into a training set, a validation set and a test set in a ratio of 8:1:1 to ensure that the training set has sufficient data for deep learning training;

[0206] Before starting the training, the parameters and hyperparameters of the algorithm are initialized, the appropriate training batch size is set according to the hardware environment, the optimizer is selected as Adam, the initial training round is set to 200, and the initial learning rate is set to 0.001; these parameters need to be optimized according to the results of multiple rounds of training until the model reaches the best effect;

[0207] After the basic parameter initialization is completed, the model training is started, and an embodiment of the training process starts from an input image Y0, and finally obtains a power worker goggle wearing detection result after a series of module processing; first, the image Y0 is processed through a multi-scale feature map generation module to extract feature maps of different scales; these feature maps are enhanced by the F-Agg network to generate multi-scale feature maps P1 to P4 containing detailed information; then, through the multi-scale feature map fusion module, these feature maps are fused into a mask feature map F mask , which is used to capture global semantic information and local details; next, the image passes through the Q2P-Cross module to generate prompt embedding feature matrices T1 to T4 using the cross attention mechanism, which provides clues for instance positioning and shape; subsequently, the B2F-Diffu module filters the generated noise boundary box to remove Gaussian noise and restore the true boundary features of the target, thereby obtaining a noise filter NF; finally, the mask feature map F mask and the noise filter NF are sent to the BMD module to generate the final instance mask feature map BM through convolution operation; the pixel point of BM is 0, indicating that the pixel at this position belongs to the background, and the pixel point of BM is 1, indicating that the pixel at this position belongs to the goggle instance; a suitable goggle wearing detection threshold is set, and in the boundary box region of the goggle instance, if the output goggle region in BM (i.e. the region with pixel point 1 in BM) is greater than the threshold, it is determined that the goggle is worn, and if the output goggle region in BM (i.e. the region with pixel point 1 in BM) is less than the threshold, it is determined that the goggle is not worn, thereby completing the goggle wearing detection task;

[0208] The training of the detection algorithm of the present application updates the parameters inside the algorithm through the back propagation of the loss function, and after obtaining the optimal network parameters, the EGSNet network training is completed; Next, the trained EGSNet network is integrated into the power construction site monitoring system; The system will collect images in real time through the camera installed in the power construction site, and transmit them to the server for processing;

[0209] In the deployment environment, ensure that the server has the necessary software library support and computing resources (such as GPU) to accelerate model inference; After the image is processed, it is sent to the trained network model, and the model uses multi-scale feature map fusion, noise filter and other technologies to accurately detect whether the power worker wears goggles; If the system detects abnormal wearing (for example, not wearing or wearing is not standardized), an alarm signal will be sent immediately to remind the staff to handle it.

[0210] Although the present application has been described above with reference to the embodiments, various modifications can be made and components therein can be replaced with equivalents without departing from the scope of the present application. In particular, features in the embodiments disclosed in the present specification can be combined with each other in any manner unless there is a structural conflict, and the combinations are not exhaustively described in the present specification only for the purpose of omitting the description and saving resources. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for detecting electrician goggles based on instance segmentation, characterized in that: The specific steps are as follows: S1: Construct the original dataset D1 of images of power workers wearing goggles; S2: Preprocess D1 to obtain a new dataset D2 of images of power workers wearing goggles. S3: Design an EGSNet network and use it to process any power worker working images in D2 to obtain the wear detection results of power workers’ goggles; S4: Load the training set and validation set in D2 into the EGSNet network for training to update the network parameters, and then use the test set in D2 as input to verify the network effect; S5: Apply the trained EGSNet network to the goggles detection task and integrate it into the power construction site monitoring system; The specific steps of S3 are as follows: S31: Construct a multi-scale feature map generation module to generate feature-enhanced multi-scale feature maps; S32: Use the multi-scale feature map fusion module to fuse P1, P2, P3 and P4 of different scales and output a mask feature map F mask ; S33: Construct the Q2P-Cross module and use the cross attention mechanism and multi-layer perceptron MLP to fuse and enhance the features of P1, P2, P3, and P4 to generate the goggles prompt embedding feature matrix. The specific steps are as follows: S331: Using a cross-attention mechanism to perform attention weight calculation on P1, P2, P3, and P4, to obtain a goggles query matrix, including a first goggles query matrix Q1, a second goggles query matrix Q2, a third goggles query matrix Q3, and a fourth goggles query matrix Q4; S332: Use a multi-layer perceptron (MLP) to enhance the features of Q1, Q2, Q3, and Q4 to generate a goggles prompt embedding matrix, including a first goggles prompt embedding matrix E1, a second goggles prompt embedding matrix E2, a third goggles prompt embedding matrix E3, and a fourth goggles prompt embedding matrix E4, which are used to guide the subsequent mask generation process. The formula is: From i =Φ mlp (Q i ),1≤i≤4 Among them, Q i represents the goggles query matrix; Φ mlp Represents a multi-layer perceptron MLP, which consists of multiple fully connected layers and ReLU activation functions; E i represents the goggles prompt embedding matrix, where E1 is the first goggles prompt embedding matrix obtained by performing a multi-layer perceptron operation on Q1, E2 is the second goggles prompt embedding matrix obtained by performing a multi-layer perceptron operation on Q2, E3 is the third goggles prompt embedding matrix obtained by performing a multi-layer perceptron operation on Q3, and E4 is the fourth goggles prompt embedding matrix obtained by performing a multi-layer perceptron operation on Q4; S333: E1, E2, E3 and E4 are transformed by sine transform sin to generate a goggles prompt embedding feature matrix, including a first goggles prompt embedding feature matrix T1, a second goggles prompt embedding feature matrix T2, a third goggles prompt embedding feature matrix T3, and a fourth goggles prompt embedding feature matrix T4. The above goggles prompt embedding feature matrix is ​​used to provide the position information and shape feature prompts of the goggles. The calculation formula is: T i =E i +that(E i ),1≤i≤4 Among them, E i represents the goggles prompt embedding matrix; sin(E i ) indicates E i Perform sine transform; T i represents the goggles prompt embedding feature matrix, where T1 is the first goggles prompt embedding feature matrix obtained by adding E1 and sin(E1), T2 is the second goggles prompt embedding feature matrix obtained by adding E2 and sin(E2), T3 is the third goggles prompt embedding feature matrix obtained by adding E3 and sin(E3), and T4 is the fourth goggles prompt embedding feature matrix obtained by adding E4 and sin(E4); S34: Construct the B2F-Diffu module to generate the noise filter NF associated with each goggle instance; S35: Use the BMD module to construct an instance mask feature map of the power worker goggles and obtain the wearing detection result of the power worker goggles.

2. The method for detecting electrician goggles based on instance segmentation according to claim 1, characterized in that: The specific steps of S1 are as follows: S11: Install surveillance cameras and cameras at power grid construction sites to capture images of power workers working under different lighting, weather, shooting angles, and postures. S12: Among all the collected images of electrician working, the images of the electrician wearing goggles correctly are selected as positive samples; S13: Images of people not wearing goggles, wearing goggles incorrectly, or wearing other glasses are used as negative samples. The number of positive and negative samples is relatively balanced, and ultimately a dataset D1 of original images of power workers wearing goggles is formed.

3. The method for detecting electrician goggles based on instance segmentation according to claim 1, characterized in that: The specific steps of S2 are as follows: S21: Perform data cleaning on the collected D1 to remove fuzzy, low-quality and duplicate samples; S22: Construct a ground truth labeled GT sample, which includes a bounding box, instance mask, and instance identification annotation data for each instance in the image. Specifically, use an annotation tool to annotate D1, annotate the bounding box of the area where the power worker wears goggles in D1, and the bounding box attributes include its location information and category information, where the category information includes two types: wearing goggles and not wearing goggles. In addition, use the annotation tool to draw instance masks for the images of the power worker wearing goggles in D1. The images of the power worker not wearing goggles in D1 do not need to draw instance masks. S23: To enhance data diversity and improve model robustness, data augmentation techniques are used to randomly rotate, crop, flip, and adjust the lighting of the images in D1 to obtain a data-enhanced dataset. S24: Segment the data set after data enhancement into a training set, a validation set, and a test set in a ratio of 8:1:1, to obtain a new data set D2 of images of power workers wearing goggles at work.

4. The method for detecting electrician goggles based on instance segmentation according to claim 1, characterized in that: The specific steps of S31 are as follows: S311: Perform a pooling operation on the image Y0∈D2 with a spatial resolution of H×W and a number of channels of C to obtain the first goggle shallow feature map Y1 with a spatial resolution of H / 2×W / 2 and a number of channels of C. Then, after two Conv1×1 convolution operations, a second goggle shallow feature map Y2 with a spatial resolution of H / 2×W / 2 and a number of channels of 2C is obtained. S312: Input Y2 into the goggles wearing detection feature extraction submodule for feature extraction to obtain a multi-scale feature map, including a first-layer feature map F1, a second-layer feature map F2, a third-layer feature map F3, and a fourth-layer feature map F4; S313: Design an F-Agg network to enhance the features of F1, F2, F3 and F4 by propagating from deep semantic features to shallow features, and obtain a multi-scale feature map of goggles with enhanced features, including the first-layer enhanced feature map P1, the second-layer enhanced feature map P2, the third-layer enhanced feature map P3 and the fourth-layer enhanced feature map P4.

5. The method for detecting electrician goggles based on instance segmentation according to claim 4, characterized in that: The specific steps of S312 are as follows: S3121: Input Y2 into the first layer Conv_Layer1 of the goggles wearing detection feature extraction submodule, perform a Conv5×5 convolution operation on Y2 using a 5×5 convolution kernel to capture local features, use batch normalization (BN) to reduce the gradient vanishing problem, and use the ReLU activation function to introduce nonlinearity and enhance the model's expressiveness, achieving preliminary feature extraction and obtaining the first-layer feature map F1 for goggles wearing detection. S3122: Input F1 to the second layer Conv_Layer2 of the goggles wearing detection feature extraction submodule, perform a Conv1×1 convolution operation on F1 using a convolution kernel of size 1×1 to adjust the number of channels, change the depth of the feature map, use BN operation and ReLU function operation to reduce gradient vanishing and introduce nonlinearity, then use a convolution kernel of size 3×3 to perform a Conv3×3 convolution operation to capture local information in the image, use a convolution kernel of size 1×1 to perform a Conv1×1 convolution operation to restore the number of channels, and perform BN operation and ReLU function operation on the feature map after channel restoration to obtain the second layer feature map F2 of goggles wearing detection; S3123: After performing residual addition on F1 and F2, the residuals are input to the third layer Conv_Layer3 of the goggles wearing detection feature extraction submodule, and a 1×1 convolution kernel is used to perform a Conv1×1 convolution operation to adjust the number of channels and change the depth of the feature map. The BN operation and the ReLU function are used to reduce the gradient disappearance and introduce nonlinearity. Then, a 3×3 convolution kernel is used to perform a Conv3×3 convolution operation to capture local information in the image. A 1×1 convolution kernel is used to perform a Conv1×1 convolution operation to restore the number of channels. The feature map after the channel restoration is subjected to a BN operation and a ReLU function operation to obtain the third layer feature map F3 for goggles wearing detection; S3124: After performing residual addition on F2 and F3, the residuals are input to the fourth layer Conv_Layer4 of the goggles wearing detection feature extraction submodule, and a 1×1 convolution kernel is used to perform a Conv1×1 convolution operation to adjust the number of channels and change the depth of the feature map. The BN operation and the ReLU function are used to reduce the gradient disappearance and introduce nonlinearity. Then, a 3×3 convolution kernel is used to perform a Conv3×3 convolution operation to capture local information in the image. A 1×1 convolution kernel is used to perform a Conv1×1 convolution operation to restore the number of channels. The feature map after the channel restoration is subjected to a BN operation and a ReLU function operation to obtain the fourth layer feature map F4 for goggles wearing detection; The specific steps of S313 are as follows: S3131: Take F4 as input. To unify the number of channels, a 1×1 convolution kernel is used to perform a Conv1×1 convolution operation on F4 to obtain the feature map F'4. A 3×3 convolution kernel is used to perform a Conv3×3 convolution operation on F'4 to enhance spatial information. A ReLU activation function is used to introduce nonlinearity to obtain the fourth-layer enhanced feature map P4 for goggles wearing detection. S3132: Perform a Conv1×1 convolution operation on F3 using a convolution kernel of size 1×1 to obtain a feature map F'3. Add the feature maps F'4, F'3, and P4 to obtain M3. Perform a Conv3×3 convolution operation on M3 using a convolution kernel of size 3×3 to obtain the third-layer enhanced feature map P3 for goggles wearing detection. S3133: Perform a Conv1×1 convolution operation on F2 using a convolution kernel of size 1×1 to obtain a feature map F'2. The feature maps M3, F'2, and P3 are added to obtain M2. A Conv3×3 convolution operation is performed on M2 using a convolution kernel of size 3×3 to obtain the second-layer enhanced feature map P2 for goggles wearing detection. S3134: Take F1 as input and perform Conv1×1 convolution to obtain feature map F'1. Add feature maps M2, F'1, and P2 to obtain M1. Perform Conv3×3 convolution on M1 to obtain the first-layer enhanced feature map P1 for goggles wearing detection. S3135: After convolution operation and feature map fusion, the multi-scale feature map required for goggles wearing detection is enhanced, and finally the multi-scale feature maps P1, P2, P3 and P4 of goggles feature enhancement are obtained; In the F-Agg network, the goggle feature enhancement for F1, F2, F3, and F4 is defined as follows: F′ i =Φ Conv1×1 (F i ),1≤i≤4 P j =Φ Conv3×3 (M j ),1≤j≤4 M k =M k+1 +P k+1 +F′ k ,1≤k≤3 Among them, F i is the multi-scale feature map of the input; Φ Conv1×1 Represents the Conv1×1 convolution operation, which is used to adjust the number of channels through convolution; F' i Represents the feature map after channel adjustment, where F'1 is the feature map after channel adjustment obtained by performing Conv1×1 convolution operation on F1, F'2 is the feature map after channel adjustment obtained by performing Conv1×1 convolution operation on F2, F'3 is the feature map after channel adjustment obtained by performing Conv1×1 convolution operation on F3, and F'4 is the feature map after channel adjustment obtained by performing Conv1×1 convolution operation on F4; Φ Conv3×3 Conv3×3 convolution operation is used to enhance spatial information, M j Represents the feature map after the goggles feature fusion, where M4 is the deepest feature map, namely F'4, M3 is the feature map obtained by fusing and adding the features of M4, P4 and F'3, M2 is the feature map obtained by fusing and adding the features of M3, P3 and F'2, and M1 is the feature map obtained by fusing and adding the features of M2, P2 and F'1; P j It is a multi-scale feature map of goggles feature enhancement. P1 is the first layer enhanced feature map obtained after performing Conv3×3 convolution operation on M1, P2 is the second layer enhanced feature map obtained after performing Conv3×3 convolution operation on M2, P3 is the third layer enhanced feature map obtained after performing Conv3×3 convolution operation on M3, and P4 is the fourth layer enhanced feature map obtained after performing Conv3×3 convolution operation on M4.

6. The method for detecting electrician goggles based on instance segmentation according to claim 1, characterized in that: The specific steps of S32 are as follows: S321: Perform a fusion operation on P1, P2, P3 and P4 to obtain global semantic information and fine local detail information, and obtain the fused fifth-layer enhanced feature map P5; S322: After performing Conv1×1 convolution operation on P5 using a convolution kernel of size 1×1, batch normalization BN operation and ReLU function operation are performed to obtain the mask feature map F mask .

7. The method for detecting electrician goggles based on instance segmentation according to claim 1, characterized in that: The specific steps of S331 are as follows: S3311: The preset Q0 is the goggle query matrix that has been initialized to zero. The dimension of Q0 is (N q ,d), where N q Represents the number of features in the query, d represents the feature dimension of the query, and the cross-attention mechanism is used to calculate the attention weight between P1 and Q0. Softmax normalization is used to obtain the attention weight matrix A1. A1 is added to Q0 to generate the first goggle query matrix Q1. S3312: Use the cross-attention mechanism to calculate the attention weights between P2 and Q1, use Softmax normalization to obtain the attention weight matrix A2, add A2 and Q1 to generate the second goggle query matrix Q2; S3313: Use the cross-attention mechanism to calculate the attention weights between P3 and Q2, use Softmax normalization to obtain the attention weight matrix A3, add A3 and Q2 to generate the third goggle query matrix Q3; S3314: Use the cross-attention mechanism to calculate the attention weights between P4 and Q3, use Softmax normalization to obtain the attention weight matrix A4, add A4 and Q3 to generate the fourth goggle query matrix Q4, the formula is: Q i =A i +Q i-1 ,1≤i≤4 Among them, A i represents the attention weight matrix calculated by the cross attention mechanism; Q i represents the goggles query matrix; A i and Q i The dimensions are (N q ,d),N q represents the number of query features; d represents the feature dimension of the query; P i is a feature map with a dimension of (H×W,d); Φ attention (P i ,Q i-1 ) represents the cross attention function, which is used to calculate the goggles query matrix Q i-1 and feature map P i Information interaction between Q 、W K and W V is a linear transformation matrix, Q i-1 and P i Project to the same feature dimension d; Yes N q A matrix of H×W rows and columns represents the attention weights; Softmax represents the activation function, which ensures that the weights are between 0 and 1 and the sum is 1, normalizing the distribution of attention weights; i and Q i-1 Add up to get Q i , can accumulate from the feature map P i The information obtained allows it to gradually learn the key features of the goggles.

8. The method for detecting electrician goggles based on instance segmentation according to claim 1, characterized in that: The specific steps of S34 are as follows: S341: Perturb the GT and enter the forward Forword stage to gradually add Gaussian noise to the GT samples to generate NBB. The specific formula of the noise perturbation process is as follows: Among them, q(x t |x t-1 ) represents the noise perturbation process, x0 represents the real image without noise, t represents the tth time of adding Gaussian noise, T represents the total number of times noise is added, represents Gaussian distribution, β t It represents the variance of Gaussian distribution, which is used to control the size of the added noise. I represents the covariance matrix of the noise, which is a unit matrix with the same size as the input data x. t or x t-1 The dimensions are consistent; S342: By sampling a Gaussian independent noise variable ∈~N(0,I) on x0, we get x t The specific formula is as follows: in, is the weight coefficient, which represents the cumulative value of the variance at each time step; ∈ represents the Gaussian independent noise variable, which is the random noise introduced in each diffusion process. As t increases, x t It is getting closer and closer to pure noise. When T→∞, x T It is a complete Gaussian noise, and the goggles noise bounding box NBB∈{x1,x2,…,x T }; S343: NBB is input into the reverse stage together with T1, T2, T3, and T4 for noise reduction, which is used to reverse the noise addition process and calculate the value from p(x t-1 |x t ) sampling, the reverse process formula is as follows: During the noise reduction process, represents Gaussian distribution; μ(x t ,t) represents the mean, ∑(x t ,t) is the covariance, the mean and covariance are optimized by minimizing the loss function and parameterized by the neural network; p(x t-1 |x t ) indicates that according to the current state x t Estimate the state x at the previous moment t-1 ,The model recovers the image features before noise addition by learning parameters, and obtains the bounding box feature BBF after noise removal in this process; S344: Send the BBF to the fully connected layer FC. After linear transformation and ReLU activation function operation, a noise filter NF related to the goggles instance is generated. The process of generating NF is as follows: NF=η(f(x t ,t)) Among them, η represents the fully connected layer, which is used to learn the mapping relationship between the denoised features and the filter, f(x t ,t) is the bounding box feature after removing noise.

9. The method for detecting electrician goggles based on instance segmentation according to claim 1, characterized in that: The specific steps of S35 are as follows: S351: Using NF to F mask Convolution is performed to extract the shape, boundaries, and other key features of the specific goggles instance, obtaining a feature map S1 with bounding box information. S1 includes the position and size information of the bounding box. Then, a Conv1×1 convolution operation of size 1×1 is used to convolve S1 to adjust the number of channels of the feature map, thereby obtaining S2. S352: Use the Sigmoid activation function to operate on S2 and constrain the output value of S2 to be between [0, 1]. Specifically: set a pixel judgment threshold H1 and binarize the output value of S2. If the output value is less than H1, it means that the pixel belongs to the background and the corresponding output value is 0; if the output value is greater than H1, it means that the pixel belongs to the goggles instance and the corresponding output value is 1. Finally, a mask feature map BM of the electrician goggles instance composed of 0 and 1 is obtained. In BM, 0 indicates that the pixel point belongs to the background area, and 1 indicates that the pixel point belongs to the goggles area. S353: Set a goggles wearing detection threshold H2. In the bounding box area of ​​the goggles instance, if the goggles area output in BM is larger than H2, it is judged that goggles are worn. If the goggles area output in BM is smaller than H2, it is judged that goggles are not worn. The goggles wearing detection result of the electrician can be obtained.

10. The method for detecting electrician goggles based on instance segmentation according to claim 1, characterized in that: The specific steps of S4 are as follows: S41: Based on D2, the EGSNet network constructed in S3 is trained and verified. First, all neural network parameters and related hyperparameters are initialized; S42: After initializing the parameters, the training set in D2 is divided into multiple batches according to the batch size, and training is performed after inputting the batches to obtain the training loss value loss of each batch. After all batches of the training set are trained, the validation set in D2 is also divided according to the batch size and input to obtain the corresponding batch loss value batch_loss; S43: During training and validation, the algorithm learns and adjusts parameters based on the corresponding loss value. The training process is repeated for multiple rounds according to the preset training rounds. When the algorithm is trained until the loss value converges, the instance segmentation algorithm training for detecting the wearing of goggles by power workers is completed. S44: After training and verification are completed, the test set preprocessed by D2 is applied, and the test data is input into the example algorithm for detecting the wearing of goggles by power workers with the optimal network parameters after training. The detection test of the wearing of goggles by power workers is performed, and the accuracy and recall rate are used as the verification indicators of the algorithm to verify the network effect; The specific steps of S5 are as follows: S51: The trained EGSNet network is used as a module in a power construction site monitoring system. The system is connected to cameras at the power construction site to monitor the wearing of goggles by power workers in real time. S52: If the device is not worn or is worn improperly, the system will issue an alarm and the relevant images will be recorded in the system so that relevant personnel can take corresponding measures.

Images