A safety wear detection method based on key points of the human body

Abstract

The present invention discloses a safety wear detection method based on human key points, which includes: using a data enhancement method to solve the detection problem of occluded personnel, using a random erasure method to increase the number of occluded samples in the model training stage, and finally realizing the detection of target personnel; using a symmetric space transformation network in a regional pose estimation algorithm (RMPE) to correct the target bounding box of personnel, improve the accuracy of pose estimation, using parameterized pose non-maximum suppression (Pose-NMS) to solve the problem of redundant poses, obtaining the accurate pose of the target personnel, extracting human key point information, and finally obtaining 17 key point coordinates; proposing a two-dimensional projection model of the human body, according to the obtained 17 key point coordinate information, using a seven-point positioning method to obtain the head area, and using a four-point positioning method to obtain the torso area; proposing a safety wear recognition method based on ResNet50-TL, extracting features from the obtained area to be detected, classifying, obtaining the safety wear situation of workers, and realizing the detection of safety wear. The method of the present invention can effectively solve the problem of safety wear detection in complex scenes such as offshore drilling platforms.

Description

Technical Field

[0001] The present application belongs to the field of computer graphics and image processing, and relates to a safety wear detection method based on key points of the human body. Background Art

[0002] With the advancement of science and technology and social development, in recent years, computer graphics and image processing technology has been increasingly applied to people's production and life, especially in various engineering fields. This is because computer vision technology has relatively simple hardware requirements and does not require contact, but has high processing accuracy and strong adaptability, so it is widely used. Especially in the field of video surveillance, computer vision technology is widely praised for its low cost and high effectiveness, and it can customize different image processing algorithms for the functions of different regulatory systems.

[0003] At present, safety behavior detection in industrial production and construction sites usually uses cameras to collect video data, and then uses manual supervision to judge the unsafe behavior and status of people or objects. However, manual supervision increases labor costs and has strong subjectivity, making it difficult to achieve standardized and objective judgments. With the development of computer technology, people try to rely on computers to automatically obtain, analyze and monitor the wearing conditions of personnel. The current helmet recognition algorithm works well in a single specific scene, but the background of the offshore platform is complex and the postures of the staff are changeable, making it difficult to ensure a high recognition accuracy rate. In addition, there are few technical studies on the detection of the dress code of the staff in the monitoring scene. The problem of safety wear detection in simple scenes has been basically solved, but most detection methods are only applicable to simple scenes and single postures, and most of them only detect safety helmets. When applied to complex scenes and multiple postures, it is difficult to perform accurate detection due to the influence of factors such as cluttered backgrounds and diverse postures, and the recognition effect drops sharply. Safety wear detection in complex scenes has become a problem that needs to be solved urgently. Summary of the invention

[0004] In order to overcome the above defects, this application proposes a safety wear detection method based on key points of the human body. The specific steps of this application are as follows:

[0005] S1, use the improved target detection algorithm to detect people in the input image, and use the target box and coordinates

[0006] The information is output in the form of information, providing input data for key point detection of the human body;

[0007] S2, based on the detection of human targets, the EMPE framework is used to extract the key point information of all human skeletons to form a key point sequence;

[0008] S3, building a two-dimensional projection model of the human body based on the information of the key points of the skeleton;

[0009] S4, based on the two-dimensional projection model of the human body, adopts the seven-point positioning method and uses the coordinate information of seven key points on the head

[0010] The head area to be detected is located based on the information;

[0011] S5 uses the four-point positioning method to locate the left shoulder, right shoulder, left ankle, and right ankle using the coordinates of the four key points.

[0012] The torso area to be tested;

[0013] S6,adopts the safety wear recognition algorithm based on ResNet50-TL,and transfers the feature extraction capability of the ResNet50 model.

[0014] S7 uses the trained head and torso regions to extract features and recognize the helmet and safety

[0015] Full clothing;

[0016] S8, output the safety wearing result of the target person;

[0017] For step S4, the head area is determined according to the coordinate information of the key points of the head. First, the left shoulder and the right ear, the left ear and the right shoulder are connected respectively to find the intersection of the two lines, and the distance between the left shoulder and the right ear, and the distance between the right shoulder and the left ear are found. The head offset direction is determined by the intersection and the line distance. The horizontal coordinate of the intersection is used as the horizontal coordinate of the center point of the head. The minimum vertical coordinate of the five key points of the left eye, the right eye, the nose, the left ear, and the right ear is used as the vertical coordinate of the center point of the head. A circle is drawn with the center point of the head as the center and 2 / 3 of the maximum value of the distance between the left shoulder and the right ear, and the distance between the right shoulder and the left ear as the radius. Then the circumscribed square of the circle is found, which is the head area to be detected. Suppose the coordinates of the left ear, the right ear, the left shoulder, and the right shoulder are Le(x1, y1), Re(x2, y2), Ls(x3, y3), and Rs(x4, y4). Then the distance between the left ear and the right shoulder is The equation of the straight line is determined to be The distance between the right ear and the left shoulder is The equation of the straight line is determined to be Find the intersection point M(x0, y0) of the straight lines L1 and L2. x0 represents the relative offset of the head. The horizontal coordinate x0 is used as the horizontal coordinate of the center point of the head. Let the coordinates of the left eye, right eye, and nose be L eye (x5,y5), R eye(x6, y6), N(x7, y7). Since the helmet is located above the head, find the key point T(x8, y8) at the top of the head key points. Use the ordinate y8 of the top key point T as the ordinate of the head center point. Draw a circle C with the center point of the head as the center and 2 / 3max{|LeRs|,|ReLs|} as the radius. Finally, the circumscribed square S of C is obtained, which is the head area to be detected.

[0018] The technical features and improvements of this application are:

[0019] For step S1, the random erasing method is used to increase the sample data of the corresponding occluded personnel, so as to train a personnel target detection model suitable for complex scenes of marine platforms; during the training process, the samples are randomly erased according to a certain probability. For a mini-batch, the probability of being randomly erased is p, and the probability of remaining unchanged is 1-p. In this process, training images with different degrees of occlusion are generated.

[0020] For step S2, a top-down regional pose estimation algorithm (RMPE) is used. RMPE includes three parts: symmetric spatial transformation network (SSTN), parameterized pose non-maximum suppression (NMS), and pose-guided region box generator (PGPG). In order to obtain a more accurate detection frame of the target area, RMPE adds a spatial symmetric transformation network (SSTN) and a parallel single-person pose detector (Parallel The spatial transformation network automatically selects the optimal ROI through 2D affine transformation to extract the most accurate human region frame. After extracting the accurate human region, the single person pose estimator (SPPE) is used to accurately estimate the pose of the target person. The spatial inverse transformation network (SDTN) maps the estimated human pose back to the original image coordinates. The parallel single person pose detector (PSPPE) is used as an additional regularization term in the training stage to avoid falling into the local optimum and further improve the effect of SSTN. Person target detection will inevitably produce redundant detection, and redundant detection will produce redundant pose estimation. RMPE uses the parameterized pose non-maximum suppression method to solve the redundant detection problem. The parameterized pose non-maximum suppression method is more efficient and accurate than the previous pose non-maximum suppression method, and does not require manual setting of parameters. Parameters can be set in a data-driven way. In order to obtain a more accurate pose, it is necessary to define pose similarity to eliminate poses that are too close and too similar. The parameterized pose non-maximum suppression defines a pose distance scale to measure pose similarity. The distance scale formula is:

[0021] d(P i ,P j |Λ)(1)

[0022] In formula (1), Λ is the parameter set of function d(·), and the elimination criterion is adopted:

[0023] f(P i ,P|Λ,η)=1[d(P i ,P j |Λ,λ)≤η] (2)

[0024] In formula (2), η is used as the threshold to determine whether to eliminate it. If d() is less than η, f(·) outputs 1, and Pi is the redundant posture. Finally, the accurate posture of the target person is obtained, and the coordinate information of 17 key points of the human body is extracted.

[0025] For step S5, the four-point positioning method is used to locate the torso area. The torso area is the part from the neck below the ankles above the ankles. Therefore, the torso area is determined by the left shoulder, the right shoulder, the left ankle, and the right ankle. The left shoulder and the left ankle, the right shoulder and the right ankle, the left ankle and the right ankle, and the left shoulder and the right shoulder are connected in sequence to generate an irregular quadrilateral bounding box. The irregular quadrilateral bounding box is corrected into a rectangular box according to the principle of the maximum side. However, because only four key points of the left shoulder, the right shoulder, the left ankle, and the right ankle are considered, the arms are ignored. Therefore, a multi-scale approach is adopted to expand the perception field. The width of the initial torso area is expanded at a ratio of 1.0 to 1.4 times, and finally the torso area bounding box to be detected is obtained.

[0026] For step S6, a safety wear recognition algorithm based on ResNet50-TL is adopted. The ResNet50, a trained convolutional neural network model structure and parameters, are used to solve the problem of safety helmet and safety clothing recognition through retraining. The ResNet50 model is selected as the pre-training model, the other network layers of ResNet50 are frozen, a fully connected layer is added, and the offshore drilling platform data set is used for training. Finally, a safety wear recognition model suitable for offshore drilling platforms is obtained.

[0027] The safety wear detection method based on key points of the human body in this application solves the problem of safety wear detection in complex scenarios such as offshore drilling platforms, and has the following advantages:

[0028] (1) The method of this application can effectively solve the problem of detecting occluded persons and realize human target detection in complex scenes;

[0029] (2) The method of the present application can solve the problem of safety wear detection in various postures. By extracting the key point information of the human body, a two-dimensional projection model of the human body is established, and the area to be detected (head area and torso area) is located. The method of the present application can also locate other areas to be detected (seat belt area, shoe area, etc.);

[0030] (3) The method of the present application utilizes convolutional neural networks and transfer learning to solve the problem of safety wear detection under small samples, avoids the work of labeling a large amount of data sets, and realizes safety wear detection in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of the safety wear detection method based on key points of the human body in this application.

[0032] Figure 2 This is a flowchart for extracting key point information of the human body based on the posture recognition algorithm in this application.

[0033] Figure 3 This is a flowchart of safety wear identification based on the ResNet50-TL algorithm in this application. DETAILED DESCRIPTION

[0034] The present application is further described in detail below with reference to the accompanying drawings and specific implementation methods:

[0035] A safety wear detection method based on key points of the human body, such as Figure 1 As shown, it is a flow chart of the safety wear detection method based on key points of the human body of the present application, and the method comprises:

[0036] S1, target personnel detection, the offshore drilling platform has a complex background and dense pipelines. Workers are often blocked by dense pipelines during construction. Therefore, the random erasing method is used to increase the corresponding occlusion sample data, so as to train a target personnel detection model suitable for complex scenes of offshore platforms and improve the detection accuracy of blocked personnel. The data sets used in training the target personnel detection model come from two aspects. One is the person pictures and their annotation files in the coco17 data set, a total of 3235 pictures, and the other is the pictures and annotation files containing workers from the offshore drilling platform site, a total of 1500 pictures. When randomly erasing the samples, there are three situations: no distinction between the target area and the background area, and the erasing process is performed on the entire image; the target frame and the background area are distinguished for erasing, and the erasing process is performed only in the target frame; the target frame and the background area are distinguished, and the erasing process is performed in the target frame and the entire image respectively. The trained target personnel detection model is used to detect the input offshore drilling platform site picture to obtain the personnel target bounding box.

[0037] S2, human key point extraction, based on the detection of the target person, the person's bounding box is used as input to extract the coordinate information of the human key points through the regional pose estimation (RMPE) algorithm. Since the accuracy of pose detection is highly dependent on the target region detection box, in order to obtain a more accurate target region detection box, the spatial transformation network is used to automatically select the optimal ROI through 2D affine transformation, thereby extracting the most accurate human region box. Affine transformation process:

[0038]

[0039] In formula (1) is the transformed coordinate, are the coordinates before transformation, θ1, θ2, θ3 are the transformation parameters.

[0040] Then the estimated human posture is mapped back to the original image coordinates using the spatial inverse transform network (SDTN), which is defined as:

[0041]

[0042] In formula (2) is the transformed coordinate, are the coordinates before transformation, and γ1, γ2, and γ3 are the transformation parameters.

[0043] Personnel target detection will inevitably produce redundant detection, which in turn will produce redundant posture estimation. RMPE uses the parametric Pose-NMS method to solve the redundant detection problem. The parametric Pose-NMS method is more efficient and accurate than the previous posture non-maximum suppression, and does not require manual setting of parameters. Parameters can be set in a data-driven manner. In order to obtain a more accurate posture, it is necessary to define posture similarity to eliminate postures that are too close or too similar. Parametric Pose-NMS defines a posture distance scale to measure posture similarity. The distance scale formula is:

[0044] d(P i ,P j |Λ)(3)

[0045] In formula (3), Λ is the parameter set of function d(·), and the elimination criterion is adopted:

[0046] f(P i ,P|Λ,η)=1[d(P i ,P j |Λ,λ)≤η] (4)

[0047] In formula (4), η is used as the threshold to determine whether to eliminate it. If d() is less than η, f(·) outputs 1, and Pi is a redundant posture. Finally, the accurate posture of the target person is obtained, and the coordinate information of 17 key points of the human body is extracted.

[0048] S3, build a two-dimensional projection model of the human body, obtain the key point coordinates of the human body through the posture recognition algorithm, save the key point coordinate information to the corresponding json file, use OpenCV encoding to read the key point information, mark the 17 key point coordinates of the human body to the corresponding target person, and finally obtain the target person with key point information.

[0049] S4, head area positioning, determine the head area according to the coordinate information of the key points of the head. First, connect the left shoulder and right ear, the left ear and right shoulder respectively, find the intersection of the two lines, and find the distance between the left shoulder and the right ear, the distance between the right shoulder and the left ear, and determine the head offset direction with the intersection and the line distance. The horizontal coordinate of the intersection is used as the horizontal coordinate of the center of the head, and the minimum vertical coordinate of the five key points of the left eye, right eye, nose, left ear, and right ear is used as the vertical coordinate of the center of the head. With the center of the head as the center, draw a circle with 2 / 3 of the maximum value of the distance between the left shoulder and the right ear, and the distance between the right shoulder and the left ear as the radius, and then find the circumscribed square of the circle, which is the head area to be detected. Suppose the coordinates of the left ear, right ear, left shoulder, and right shoulder are Le(x1,y1), Re(x2,y2), Ls(x3,y3), and Rs(x4,y4), then the distance between the left ear and the right shoulder is The equation of the straight line is determined to be The distance between the right ear and the left shoulder is The equation of the straight line is determined to be The intersection point M(x0,y0) of the straight lines L1 and L2 is obtained. x0 represents the relative offset of the head. The horizontal coordinate x0 is used as the horizontal coordinate of the center point of the head. The coordinates of the left eye, right eye, and nose are L eye (x5,y5), R eye (x6, y6), N(x7, y7). Since the helmet is located above the head, find the key point T(x8, y8) at the top of the head key points. Use the ordinate y8 of the top key point T as the ordinate of the head center point. Draw a circle C with the center point of the head as the center and 2 / 3max{|LeRs|,|ReLs|} as the radius. Finally, the circumscribed square S of C is obtained, which is the head area to be detected.

[0050] S5, torso area positioning. In order to obtain the safety clothing area, the four-point positioning method is used to locate the torso area. The torso area is the part from the neck below the ankles above. Therefore, the torso area is determined by the left shoulder, right shoulder, left ankle, and right ankle. The left shoulder and left ankle, the right shoulder and right ankle, the left ankle and right ankle, and the left shoulder and right shoulder are connected in sequence to generate an irregular quadrilateral bounding box. The irregular quadrilateral bounding box is corrected into a rectangular box according to the principle of maximum side. However, because only the four key points of the left shoulder, right shoulder, left ankle, and right ankle are considered, the arms are ignored. Therefore, a multi-scale approach is adopted to expand the perception field. The width of the initial torso area is expanded at a ratio of 1.0 to 1.4 times, and finally the torso area bounding box to be detected is obtained.

[0051] S6, based on the ResNet50-TL safety wear recognition algorithm, uses the ResNet50, a trained convolutional neural network model structure and parameters, to solve the problem of helmet and safety clothing recognition through retraining. The ResNet50 model is selected as the pre-training model, the other network layers of ResNet50 are frozen, a fully connected layer is added, and the offshore drilling platform data set is used for training, and finally a safety wear recognition model suitable for offshore drilling platforms is obtained.

[0052] S7, extracts and identifies features of the area to be detected. First, input an image of a helmet or safety clothing into the feature extraction model, and then use the convolution layer and pooling layer in the ResNet50 model to extract features, obtain the image features of the helmet and safety clothing, and represent them with a 2048-dimensional vector. At the same time, in order to avoid the time cost of processing the image multiple times, a cache file is set to save the image features; the obtained image features are used as the input of the new neural network, and then a new Softmax function is trained for the neural network to classify the image.

[0053] S8, outputs the safety wearing results of the target personnel. There are four types of results, namely, the safety helmet and safety clothing are worn normally (WHV), only wearing a safety helmet (WH), only wearing safety clothing (WV), and neither wearing a safety helmet nor wearing safety clothing (W).

[0054] In summary, the safety wear recognition method based on human key points solves the problem of safety wear detection in complex scenarios such as offshore drilling platforms in a big data environment. The proposed two-dimensional projection model of the human body can also locate other areas to be detected, which can be selected according to different needs. In addition, the safety wear recognition algorithm based on ResNet50-TL avoids the problem of large-scale data set annotation.

[0055] Although the content of the present application has been described in detail through the above preferred embodiments, it should be appreciated that the above description should not be considered as a limitation of the present application. After reading the above content, it will be apparent to those skilled in the art that various modifications and substitutions of the present application can be made. Therefore, the protection scope of the present application should be limited by the appended claims.

Claims

1. A safety wear detection method based on key points of the human body, the specific steps include: S1, uses the improved target detection algorithm to detect human targets in the input image, and outputs it in the form of target boxes and coordinate information to provide input data for human key point detection; S2, based on the detection of human targets, the EMPE framework is used to extract the key point information of all human skeletons to form a key point sequence; S3, building a two-dimensional projection model of the human body based on the information of the key points of the skeleton; S4, based on the two-dimensional projection model of the human body, a seven-point positioning method is used to locate the head area to be detected using the coordinate information of seven key points of the head; S5, using the four-point positioning method, uses the coordinates of the four key points of left shoulder, right shoulder, left ankle, and right ankle to locate the torso area to be detected; S6, uses a safety wear recognition algorithm based on ResNet50-TL, and transfers the feature extraction capability of the ResNet50 model; S7, using the trained head area and torso area to be detected to extract features and identify the helmet and safety clothing; S8, output the safety wearing result of the target person; For step S4, the head area is determined according to the coordinate information of the key points of the head. First, the left shoulder and the right ear, the left ear and the right shoulder are connected respectively to find the intersection of the two lines, and the distance between the left shoulder and the right ear, and the distance between the right shoulder and the left ear are found. The head offset direction is determined by the intersection and the line distance. The horizontal coordinate of the intersection is used as the horizontal coordinate of the center point of the head. The minimum vertical coordinate of the five key points of the left eye, the right eye, the nose, the left ear, and the right ear is used as the vertical coordinate of the center point of the head. A circle is drawn with the center point of the head as the center and 2 / 3 of the maximum value of the distance between the left shoulder and the right ear, and the distance between the right shoulder and the left ear as the radius. Then the circumscribed square of the circle is found, that is, the head area to be detected. Suppose the coordinates of the left ear, the right ear, the left shoulder, and the right shoulder are Le(x1, y1), Re(x2, y2), Ls(x3, y3), and Rs(x4, y4). Then the distance between the left ear and the right shoulder is The equation of the straight line is determined to be The distance between the right ear and the left shoulder is The equation of the straight line is determined to be Find the intersection point M(x0,y0) of the straight lines L1 and L2. x0 represents the relative offset of the head. The horizontal coordinate x0 is used as the horizontal coordinate of the center point of the head. Let the coordinates of the left eye, right eye, and nose be L eye (x5,y5), R eye (x6, y6), N(x7, y7). Since the helmet is located above the head, find the key point T(x8, y8) at the top of the head key points. Use the ordinate y8 of the top key point T as the ordinate of the head center point. Draw a circle C with the center point of the head as the center and 2 / 3max{|LeRs|,|ReLs|} as the radius. Finally, the circumscribed square S of C is obtained, which is the head area to be detected.

2. A safety wear detection method based on key points of the human body according to claim 1, characterized in that: Step S1, by using the random erasing method, add the sample data of the corresponding occluded personnel, so as to train a personnel target detection model suitable for complex scenes of marine platforms; during the training process, the samples are randomly erased according to the preset probability. For a mini-batch, the probability of being randomly erased is p, and the probability of remaining unchanged is 1-p. In this process, training images with different occlusion degrees are generated.

3. The method for detecting safety wear based on key points of the human body according to claim 1, characterized in that: For step S2, a top-down regional pose estimation algorithm (RMPE) is used. RMPE includes three parts: symmetric spatial transformation network (SSTN), parameterized pose non-maximum suppression (Pose-NMS), and pose-guided region box generator (PGPG). In order to obtain a more accurate detection frame of the target area, RMPE adds a spatial symmetric transformation network (SSTN) and a parallel single-person pose detector (Parallel The spatial transformation network automatically selects the optimal ROI through 2D affine transformation to extract the most accurate human region frame. After extracting the accurate human region, the single person pose estimator (SPPE) is used to accurately estimate the pose of the target person. The spatial inverse transformation network (SDTN) maps the estimated human pose back to the original image coordinates. The parallel single person pose detector (PSPPE) is used as an additional regularization term in the training stage to avoid falling into the local optimum and further improve the effect of SSTN. Person target detection will inevitably produce redundant detection, and redundant detection will produce redundant pose estimation. RMPE uses the parameterized pose non-maximum suppression method (Pose-NMS) to solve the redundant detection problem. Pose-NMS is more efficient and accurate than the previous pose non-maximum suppression, and does not require manual setting of parameters. Parameters can be set in a data-driven way. , In order to obtain a more accurate posture, it is necessary to define posture similarity to eliminate postures that are too close or too similar. Pose-NMS defines a posture distance scale to measure posture similarity. The distance scale formula is: d(P i ,P j |Λ)(1)In formula (1), Λ is the parameter set of function d(·), and the elimination criterion is: f(P i ,P|Λ,η)=1[d(P i ,P j |Λ,λ)≤η] (2) In formula (2), η is used as the threshold to determine whether to eliminate. If d() is less than η, f(·) outputs 1, P i The redundant posture finally obtains the accurate posture of the target person and extracts the coordinate information of 17 key points of the human body.

4. The method for detecting safety wear based on key points of the human body according to claim 1, characterized in that: In step S5, the torso region is located using a four-point positioning method. The torso region is the part from the neck to the ankles. The torso region is determined by the left shoulder, the right shoulder, the left ankle, and the right ankle. The left shoulder and the left ankle, the right shoulder and the right ankle, the left ankle and the right ankle are connected in sequence. The right ankle, left shoulder and right shoulder generate an irregular quadrilateral bounding box, which is corrected into a rectangular box according to the principle of maximum side. Only four key points, namely the left shoulder, right shoulder, left ankle and right ankle, are considered, and the arms are ignored. A multi-scale approach is adopted to expand the receptive field, and the width of the initial torso area is expanded at a ratio of 1.0 to 1.4, and finally the torso area bounding box to be detected is obtained.

5. The method for detecting safety wear based on key points of human body according to claim 1, characterized in that: For step S6, based on the safety wear recognition algorithm of ResNet50-TL, the convolutional neural network model structure and parameters of ResNet50 that have been trained are used to solve the problem of safety helmet and safety clothing recognition through retraining. The ResNet50 model is selected as the pre-training model, and the other network layers of ResNet50 are frozen. A fully connected layer is added, and the labeled offshore drilling platform data set is used for training, and finally a safety wear recognition model suitable for offshore drilling platforms is obtained.

Images