A worker safety helmet detection method and device based on feature fusion

Abstract

The application discloses a worker safety helmet detection method based on feature fusion, and comprises the following steps: acquiring a sample image; acquiring first features of a target safety helmet contained in the sample image through a panoramic segmentation attention mechanism; acquiring second features of the target safety helmet contained in the sample image through a generative model; fusing the first features and the second features to obtain fused features; training and testing a worker safety helmet detection model by using the fused features; and detecting a worker safety helmet in an image to be recognized by using the model. The application solves the problem of high labor cost caused by limited energy of safety control personnel.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a worker safety helmet detection method and device based on feature fusion. BACKGROUND

[0002] In the process of power construction, power operating personnel are required to wear safety helmets. However, in recent years, substation production safety incidents have occurred frequently, and the root cause is that operating personnel violate power safety specifications for operation, one of which is not wearing safety helmets according to specifications.

[0003] As known, safety helmets are important safety protection tools for substation operating personnel, and currently rely on safety control personnel for supervision, which is high in labor cost. SUMMARY

[0004] To solve the above problems, the present application provides a worker safety helmet detection method based on feature fusion, comprising:

[0005] obtaining a sample image;

[0006] obtaining a first feature of a target safety helmet contained in the sample image through a panoramic segmentation attention mechanism;

[0007] obtaining a second feature of the target safety helmet contained in the sample image through a pre-constructed extremely difficult negative sample generation model for the sample image with feature recognition error of the target safety helmet;

[0008] fusing the first feature and the second feature to obtain a fusion feature, and using the fusion feature to train and test the worker safety helmet detection model;

[0009] using the model to detect a worker safety helmet for an image to be recognized.

[0010] Preferably, the sample image is obtained, comprising:

[0011] obtaining an image of a worker work area through a camera device, and taking the image as a sample image.

[0012] Preferably, before the step of extracting the feature of the sample image through the panoramic segmentation attention mechanism, the method further comprises:

[0013] dividing the sample image into a plurality of grids uniformly, and dynamically allocating a weight to each grid;

[0014] performing object detection division on the plurality of grids, extracting a grid containing the target safety helmet, and obtaining an attribute value of the grid of the safety helmet, and dynamically allocating a weight to the grid of the target safety helmet.

[0015] Preferably, the first feature of the sample image containing the target safety helmet is extracted through the panoramic segmentation attention mechanism, including:

[0016] The sample image is segmented through the panoramic segmentation attention mechanism.

[0017] The foreground region and the background region of the grid containing the target safety helmet of the sample image are obtained through the convolutional neural network.

[0018] Features of the foreground region and the background region are extracted, and the features are taken as the first feature of the sample image containing the target safety helmet.

[0019] Preferably, the loss function of the negative sample generation model is specific to:

[0020]

[0021] Wherein, [x] + = max(x, 0), I' represents the image with the highest similarity to the text description C in each batch of training samples, C' represents the text description with the highest similarity to the image I in each batch of training samples, C f represents the extremely difficult negative sample text description generated corresponding to C, s(I, C) represents the similarity between the image I and the matching text description C, s(I, C') represents the similarity between the image I and the non-matching text description C', s(I', C) represents the similarity between the text description C and the non-matching image I', and s(I, C f) represents the similarity between the image I and the extremely difficult negative sample text description C f , the loss function requires that the similarity of the matching image and text is higher than the threshold α, and the similarity of the matching image and text is higher than the threshold α f .

[0022] Preferably, the first feature and the second feature are fused to obtain a fusion feature, and the worker safety helmet detection model is trained and tested using the fusion feature, including:

[0023] The first feature and the second feature are input into the fusion layer of the convolutional neural network for fusion to obtain a fusion feature, and the worker safety helmet detection model is trained and tested through the fusion feature.

[0024] Preferably, after the step of using the model to detect the worker safety helmet of the image to be recognized, it further includes:

[0025] When the worker does not wear the safety helmet, an alarm is issued.

[0026] The application also provides a worker safety helmet detection device based on feature fusion, including:

[0027] a sample image acquisition unit, configured to acquire a sample image;

[0028] a first feature acquisition unit, configured to acquire a first feature of the sample image containing a target safety helmet through a panoramic segmentation attention mechanism;

[0029] a second feature acquisition unit, configured to acquire a second feature of the sample image containing the target safety helmet through a pre-constructed negative sample generation model for the sample image with feature recognition error of the target safety helmet;

[0030] a fusion unit, configured to fuse the first feature and the second feature to acquire a fusion feature, and use the fusion feature to train and test the worker safety helmet detection model;

[0031] a detection unit, configured to use the model to detect a worker safety helmet in an image to be recognized.

[0032] Preferably, the method further comprises:

[0033] a grid division unit, configured to uniformly divide the sample image into a plurality of grids, and dynamically allocate a weight to each grid;

[0034] an object detection division unit, configured to perform object detection division on the plurality of grids, extract a grid containing the target safety helmet, obtain an attribute value of the grid of the safety helmet, and dynamically allocate a weight to the grid of the target safety helmet.

[0035] Preferably, the first feature acquisition unit comprises:

[0036] a segmentation sub-unit, configured to segment the sample image through the panoramic segmentation attention mechanism;

[0037] a foreground region and background region acquisition sub-unit, configured to acquire a foreground region and a background region of the grid of the sample image containing the target safety helmet through a convolutional neural network;

[0038] the first feature acquisition unit, configured to extract features of the foreground region and the background region, and use the features as the first feature of the sample image containing the target safety helmet. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a flowchart of a worker safety helmet detection method based on feature fusion provided by an embodiment of the present application;

[0040] Figure 2 is a schematic diagram of a uniform grid division method related to an embodiment of the present application;

[0041] Figure 3is a schematic diagram of an object detection division method related to an embodiment of the present application;

[0042] Figure 4 is a structural schematic diagram of a feature fusion layer related to an embodiment of the present application;

[0043] Figure 5 is a detection result schematic diagram of a worker safety helmet detection method based on feature fusion related to an embodiment of the present application;

[0044] Figure 6 is a schematic diagram of a worker safety helmet detection device based on feature fusion provided by an embodiment of the present application. DETAILED DESCRIPTION

[0045] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced in a variety of ways beyond the specific details set forth herein without departing from the scope of the present application, and it is understood that the present application is intended to cover all modifications of the application falling within the scope of the application. Accordingly, the application is not limited to the specific details described below.

[0046] Figure 1 is a flowchart of a worker safety helmet detection method based on feature fusion provided by an embodiment of the present application, and the following Figure 1 The method provided by the present application is described in detail.

[0047] Step S101, acquiring a sample image.

[0048] An image of a worker's work area is acquired by a camera device, and the image is taken as a sample image. Specifically, a real-time image of the work area can be acquired by a camera, and the image can be captured by frame-by-frame analysis.

[0049] Step S102, acquiring a first feature of a target safety helmet contained in the sample image by a panoramic segmentation attention mechanism.

[0050] The sample image is uniformly divided into a plurality of grids by uniform grid division, and each grid is dynamically assigned a weight. The plurality of grids are subjected to object detection division, the grid containing the target safety helmet is extracted, the attribute value of the safety helmet grid is obtained, and the target safety helmet grid is dynamically assigned a weight.

[0051] Uniform grid division: the images in the data set are subjected to convolution, pooling and other operations, and are converted into n*m*l grid features in the last convolution layer. The images are divided into n*m grid regions in space, and the feature dimension of each image is l, as shown in Figure 2 The attention mechanism dynamically assigns weights to the n*m regions.

[0052] Object detection division, divide the image in the data set, divide the size and position of the detected object, each detection region contains p attribute values, which are x, y, h, w, and c, respectively. The ROI align operation is used to select the corresponding features of the region on the feature map. By setting the confidence value less than the threshold value β, the background region can be filtered out, as shown in Figure 3 The weights are dynamically allocated.

[0053] The panoramic segmentation attention mechanism is used to segment the sample image. The foreground region and the background region of the grid containing the target safety cap of the sample image are obtained through a convolutional neural network. The features of the foreground region and the background region are extracted, and the features are used as the first features of the sample image containing the target safety cap.

[0054] The panoramic segmentation attention mechanism is used to divide the image in the data set. First, the foreground and background regions are obtained through panoramic segmentation, and then the features of the segmentation region are extracted through a fully convolutional neural network. The mask value Q is set to multiply the segmentation region and the feature map, so that the features of the irrelevant region are set to P.

[0055] In step S103, the feature recognition error sample image of the target safety cap is obtained through the pre-constructed extremely difficult negative sample generation model, and the second feature of the sample image containing the target safety cap is obtained.

[0056] The feature map fusion safety cap detection method based on panoramic segmentation attention mechanism and extremely difficult negative sample mining, wherein the extremely difficult negative sample mining includes extremely difficult negative sample generation and extremely difficult negative sample generation model structure.

[0057] The extremely difficult negative sample generation takes the data set as input and performs word replacement on the images that do not appear in the annotation file.

[0058] The extremely difficult negative sample generation model structure includes the loss function of the extremely difficult negative sample generation model, which is specifically:

[0059]

[0060] wherein, [x] + = max(x, 0), I' represents the image with the highest similarity to the text description C in each batch of training samples, C' represents the text description with the highest similarity to the image I in each batch of training samples, C f represents the generated extremely difficult negative sample text description corresponding to C, s(I, C) represents the similarity between image I and the matching text description C, s(I, C') represents the similarity between image I and the unmatched text description C', s(I', C) represents the similarity between the text description C and the unmatched image I', and s(I, C f) represents the similarity between image I and the extremely difficult negative sample text description C fsimilarity of the matched image-text pair is higher than a threshold value a, and the similarity of the matched image-text pair is higher than the similarity of the very difficult negative sample and the image by a threshold value a f .

[0061] Step S104, fusing the first feature and the second feature to obtain a fused feature, and using the fused feature to train and test the worker safety helmet detection model.

[0062] The first feature and the second feature are input into a fusion layer of a convolutional neural network for fusion to obtain a fused feature, and the worker safety helmet detection model is trained and tested through the fused feature.

[0063] The feature fusion layer mixes the original features into features with stronger representation ability through a series of operations. The structure of the feature fusion layer is as shown in Figure 4 .

[0064] Step S105, using the model to detect the worker safety helmet of an image to be recognized.

[0065] Through the model, when recognizing that the worker does not wear a safety helmet, an alarm is issued.

[0066] The specific application implementation is as follows:

[0067] Step 1: using a camera to obtain real-time images of a work area, the camera obtains real-time images of the work area, and frame-by-frame analysis is performed;

[0068] Step 2: using a panoramic segmentation attention mechanism, using CNN (convolutional neural network) feature extraction, using a deep learning neural network to train a power worker safety helmet detection model;

[0069] Step 3: using samples to test the model, mining the samples that are incorrectly recognized and retraining the recognition model to extract features;

[0070] Step 4: using feature fusion to make the semantic information representation ability of the model stronger, fusing the features of steps 2 and 3, and using a deep learning neural network to train a power worker safety helmet detection model;

[0071] Step 5: using the model to detect the power worker safety helmet, using the images captured in step 1 as input, the model performs detection, and outputs the results, as shown in Figure 5 , if the system finds an anomaly, an alarm measure will be taken.

[0072] using a camera to obtain real-time images of a work area, the camera obtains real-time images of the work area;

[0073] The image is panoramic segmentation attention mechanism, very difficult negative sample mining, feature fusion, the fusion feature is sent into the deep neural network training, obtains the model and detects the safety hat.

[0074] The panoramic segmentation attention mechanism includes uniform grid division, object detection division and panoramic segmentation attention mechanism.

[0075] Preferably, the uniform grid division is that the image collected in step 1 is subjected to convolution, pooling and other operations, and then converted into a grid feature with a size of 14*14*2048 through the last convolution layer. The image is divided into 14*14 grid regions in space, and the feature dimension of each image is 2048. Then, the attention mechanism dynamically allocates weights to the 14*14 regions, and the dynamic allocation is as follows:

[0076]

[0077] α t =softmax(α t )

[0078]

[0079] wherein W α , W αν , W αh are weight matrices corresponding to the dimensions, is the score of the i-th region at time t, is the normalized weight of the i-th region at time t, t is composed of the feature of each region and the corresponding weighted sum of weights, is the hidden state of the Attention LSTM (attention mechanism recurrent neural network) at time t,

[0080] tanh and softmax are activation functions and their expressions are as follows:

[0081]

[0082] Preferably, the object detection division divides the image in the data set into detection objects, and divides the image according to the size and position of the detection object, each detection region contains 5 attribute values, and the attribute values are normalized, and the attribute values are the horizontal coordinate x (0, 1), the vertical coordinate y (0, 1), the length h (0, 1), the width w (0, 1) and the detection confidence c = 0.5. The ROIalign operation is used to select the corresponding feature of the region on the feature map through the x, y, w, h of the center position of the region, and the background region can be filtered out by the confidence less than the threshold value β = 0.5. Since the distance between the safety helmet and the camera is not the same, the attention is dynamically allocated to these regions in the attention allocation process, and the allocation is the same as the uniform grid division.

[0083] Preferably, the panoramic segmentation attention mechanism, the image in the data set, is first subjected to panoramic segmentation to obtain a foreground and a background region, and then is sent into a full convolutional neural network to extract features of the segmented region. Each region corresponds to a feature, the region of the safety helmet is segmented from the image, and the features of the full image are extracted by using the CNN (for example, taking one channel), and then the region is rescaled to the size of the feature map. The segmentation mask takes a value Q = 0 or 1, and the rescaled segmented region is directly multiplied with the feature map, so that the features of the irrelevant region are set to P = 0, while the features of the safety helmet region are kept. Therefore, the segmented regions are all processed in the same way to extract features from the image.

[0084] The feature map fusion safety helmet detection method based on the panoramic segmentation attention mechanism and the extremely difficult negative sample mining, wherein the extremely difficult negative sample mining comprises an extremely difficult negative sample generation and an extremely difficult negative sample generation model structure.

[0085] Preferably, the extremely difficult negative sample generation takes the data set as input, and replaces the words in the image not appearing in the annotation file with the same type of words. The same type of words refer to words belonging to the same category as the replaced words, for example, safety helmet and baseball cap both belong to hat, and the replaced words are (Snow cap, rain cap, sun cap, safety cap, dust cap, sleeping cap, work cap, travel cap and top hat, straw).

[0086] Preferably, the extremely difficult negative sample generation model structure has a loss function of the model structure.

[0087]

[0088] wherein, [x] + = max(x, 0), I' is an image in the first batch of training samples with the highest similarity to the textual description C, C' represents a textual description in the first batch of training samples with the highest similarity to the image I, and C fs(I,C) represents the similarity between image I and the matched text description C, then s(I,C) = 0.02, s(I,C') represents the similarity between image I and the unmatched text description C', then s(I,C') = 0.01, s(I',C) represents the similarity between the text description C and the unmatched image I', then s(I',C) = 0.05, s(I,C f) represents the similarity between image I and the very difficult negative sample text description C f , then s(I,Cf) = 0.03, the loss function requires that the similarity of the matched image-text is higher than the threshold value a, the value range of a is (0, 0.5), the value of a in this experiment is 0.2, and the similarity of the matched image-text is required to be higher than the threshold value a f , the value range of a is (0, 0.5), the value of a in this experiment is 0.2, and the similarity of the matched image-text is required to be higher than the threshold value a f , the value range of a is (0, 0.5), the value of a in this experiment is 0.2, and the similarity of the matched image-text is required to be higher than the threshold value a

[0089] The feature fusion safety hat detection method based on the panoramic segmentation attention mechanism and the very difficult negative sample mining, wherein the feature fusion includes a feature fusion layer.

[0090] The preferred feature fusion layer establishes the channel connection between different features, and the features extracted in steps 2 and 3 are fused, and the two are input into the feature fusion layer in parallel. The object detection feature Idef is first multiplied by the transposed image-text matching feature Icnn, and then normalized by the softmax function. The spliced vector has two paths: one path is directly output, and the other path is multiplied by the feature obtained by the softmax after dimension conversion by full connection, and then converted to the original dimension by full connection and added to the first path.

[0091] The feature obtained after addition is sent to a deep learning neural network, and a model is trained. The trained model is used for power worker safety hat detection. The system takes alarm measures when an anomaly is found.

[0092] In this embodiment, the overall recognition rate in the work scene is 96%.

[0093] Based on the same inventive concept, the application also provides a worker safety hat detection device 600 based on feature fusion, as shown in Figure 6 , comprising:

[0094] A sample image acquisition unit 610 is configured to acquire a sample image.

[0095] A first feature acquisition unit 620 is configured to acquire a first feature of a target safety hat contained in the sample image by a panoramic segmentation attention mechanism.

[0096] The second feature acquisition unit 630 is configured to acquire, by using a pre-constructed negative sample generation model, a second feature of the sample image containing the target safety helmet.

[0097] The fusion unit 640 is configured to fuse the first feature and the second feature to obtain a fusion feature, and use the fusion feature to train and test the worker safety helmet detection model.

[0098] The detection unit 650 is configured to use the model to detect a worker safety helmet in a to-be-recognized image.

[0099] Preferably, the method further comprises:

[0100] The grid division unit is configured to uniformly divide the sample image into a plurality of grids, and dynamically allocate a weight to each grid.

[0101] The object detection division unit is configured to perform object detection division on the plurality of grids, extract a grid containing the target safety helmet, obtain an attribute value of the grid of the safety helmet, and dynamically allocate a weight to the grid of the target safety helmet.

[0102] Preferably, the first feature acquisition unit comprises:

[0103] The segmentation sub-unit is configured to segment the sample image by using a panoramic segmentation attention mechanism.

[0104] The foreground region and background region acquisition sub-unit is configured to acquire, by using a convolutional neural network, a foreground region and a background region of the grid of the sample image containing the target safety helmet.

[0105] The first feature acquisition unit is configured to extract features of the foreground region and the background region, and use the features as the first feature of the sample image containing the target safety helmet.

[0106] The method and device for detecting a worker safety helmet based on feature fusion provided in the application use a feature map fusion and a convolutional neural network to detect a worker safety helmet, detect an anomaly, and timely alarm, which can effectively avoid the occurrence of a safety accident and provide protection for the life safety of workers. The method solves the problem of high labor cost caused by limited energy of safety control personnel.

[0107] Those skilled in the art will readily understand that the above description is only preferred embodiments of the application and is not intended to limit the application. Any modification, equivalent replacement, and improvement within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A worker safety hat detection method based on feature fusion, characterized in that, The method comprises the following steps: obtaining a sample image; obtaining a first feature of the sample image containing a target safety helmet through panoramic segmentation attention mechanism; obtaining a second feature of the sample image containing the target safety helmet through a pre-constructed extremely difficult negative sample generation model for the sample image with incorrect feature recognition of the target safety helmet; fusing the first feature and the second feature to obtain a fusion feature, and using the fusion feature to train and test the worker safety helmet detection model; detecting a worker safety helmet in an image to be recognized using the model; extracting a first feature of the sample image containing a target safety helmet through panoramic segmentation attention mechanism, comprising: segmenting the sample image through panoramic segmentation attention mechanism; obtaining a foreground region and a background region of a grid containing the target safety helmet of the sample image through a convolutional neural network; extracting features of the foreground region and the background region as the first feature of the sample image containing the target safety helmet; the loss function of the negative sample generation model, specifically: wherein [x] + = max(x, 0), I' represents the image with the highest similarity to the textual description C in each batch of training samples, C' represents the textual description with the highest similarity to the image I in each batch of training samples, C f represents the generated extremely difficult negative sample textual description corresponding to C, s(I, C) represents the similarity between the image I and the matched textual description C, s(I, C') represents the similarity between the image I and the unmatched textual description C', s(I', C) represents the similarity between the textual description C and the unmatched image I', and s(I, C f) represents the similarity between the image I and the extremely difficult negative sample textual description C f , the loss function requires that the similarity of the matched image-text is higher than the threshold value a, and that the similarity of the matched image-text is higher than the threshold value a f than the similarity of the extremely difficult negative sample and the image.

2. The method of claim 1, wherein, obtaining a sample image, comprising: obtaining an image of a worker working area through a camera device, and taking the image as a sample image.

3. The method of claim 1, wherein, Before the step of extracting the feature of the sample image through the panoramic segmentation attention mechanism, the method further comprises: dividing the sample image into a plurality of grids uniformly, and dynamically assigning weights to each grid; performing object detection division on the plurality of grids, extracting a grid containing the target safety helmet, obtaining attribute values of the grid of the safety helmet, and dynamically assigning weights to the grid of the target safety helmet.

4. The method of claim 1, wherein, fusing the first feature and the second feature to obtain a fusion feature, and using the fusion feature to train and test the worker safety helmet detection model, comprising: inputting the first feature and the second feature into a fusion layer of a convolutional neural network to fuse and obtain a fusion feature; and training and testing the worker safety helmet detection model through the fusion feature.

5. The method of claim 1, wherein, After the step of detecting a worker safety helmet in an image to be recognized using the model, the method further comprises: issuing an alarm when a worker does not wear a safety helmet.

6. A worker safety hat detection device based on feature fusion, characterized in that, The method comprises the following steps: a sample image acquisition unit is configured to obtain a sample image; a first feature acquisition unit is configured to obtain a first feature of the sample image containing a target safety helmet through panoramic segmentation attention mechanism; a second feature acquisition unit is configured to obtain a second feature of the sample image containing the target safety helmet through a pre-constructed negative sample generation model for the sample image with incorrect feature recognition of the target safety helmet; a fusion unit is configured to fuse the first feature and the second feature to obtain a fusion feature, and use the fusion feature to train and test the worker safety helmet detection model; a detection unit is configured to detect a worker safety helmet in an image to be recognized using the model; the first feature acquisition unit comprises: a segmentation subunit is configured to segment the sample image through panoramic segmentation attention mechanism; a foreground region and a background region acquisition subunit is configured to obtain a foreground region and a background region of a grid containing the target safety helmet of the sample image through a convolutional neural network; The first feature acquisition unit is configured to extract features of the foreground region and the background region, and take the features as first features of the sample image containing the target safety cap; The loss function of the negative sample generation model is specifically as follows: wherein [x] + = max(x, 0), I' represents the image with the highest similarity to the textual description C in each batch of training samples, C' represents the textual description with the highest similarity to the image I in each batch of training samples, C f represents the generated extremely difficult negative sample textual description corresponding to C, s(I, C) represents the similarity between the image I and the matched textual description C, s(I, C') represents the similarity between the image I and the unmatched textual description C', s(I', C) represents the similarity between the textual description C and the unmatched image I', and s(I, C f) represents the similarity between the image I and the extremely difficult negative sample textual description C f , the loss function requires that the similarity of the matched image-text is higher than a threshold value a, and that the similarity of the matched image-text is higher than the threshold value a f than the similarity of the extremely difficult negative sample and the image.

7. The apparatus of claim 6, wherein, Further comprising: The grid division unit is configured to uniformly divide the sample image into a plurality of grids, and dynamically allocate weights to each grid. The object detection division unit is configured to perform object detection division on the plurality of grids, extract a grid containing the target safety cap, obtain an attribute value of the safety cap, and dynamically allocate weights to the grid of the target safety cap.

Images