Building inspection robot protection wear identification method based on human body posture detection

Abstract

The application is a building inspection robot protective wear identification method based on human body posture detection. Firstly, the key points related to protective wear are selected through human body posture recognition, and each key point is converted from coordinate form to image form by pixel point filling to generate a human body posture heat map. Secondly, a protective wear identification model is constructed, which includes a first level neural network, a second level neural network and a connection module between the two levels of neural networks. Finally, the protective wear identification model is trained, and the trained protective wear identification model is transplanted into the building inspection robot to intelligently identify whether the construction workers wear gloves, reflective clothing and safety helmets, output the category, confidence and identification frame of various protective tools, and correct the reflective clothing identification frame and safety helmet confidence. The key points are used for positioning the protective tools, considering the position correlation between the protective tools and the human body wearing parts, and avoiding the interference of similar objects in the image on the identification result.

Description

Technical Field

[0001] This invention belongs to the field of intelligent detection technology for protective wear, specifically relating to a method for identifying protective wear on building inspection robots based on human posture detection. Background Technology

[0002] In construction sites and vocational colleges' training and assessment of special construction workers, workers are required to wear protective gear such as safety helmets, reflective vests, and gloves to ensure their safety. Traditional technologies use RFID to monitor the wearing of protective gear, but this method is mainly used for safety helmet detection and has low application flexibility and intelligence. With the development of technology, image recognition-based target detection methods are increasingly being used for the detection of protective gear on construction sites.

[0003] Chinese patent application number 201811032872.0 discloses a video-based automatic identification method and alarm system for the wearing status of work clothes and hats. This method segments the detected human target into head and torso portions. Using the Lab color space, it intelligently analyzes whether the head is wearing a safety helmet and whether the torso is wearing work clothes, thus achieving automatic identification of work clothes and hats. However, this method can only identify the wearing status of work clothes and hats, but cannot identify whether the worker is wearing gloves.

[0004] Zhao Rui et al. modified the YOLO neural network structure and trained the model parameters through transfer learning, making the neural network more suitable for helmet wearing detection and achieving highly reliable safety protection recognition (see Zhao Rui, Liu Hui, Liu Peilin et al. Helmet detection algorithm based on improved YOLOv5s [J]. Journal of Beijing University of Aeronautics and Astronautics, 2021). Cheng Huanxin et al. improved the YOLO neural network to simultaneously identify whether a helmet and reflective vest are worn (see Cheng Huanxin, Jiang Zeqin, Cheng Li et al. Helmet and reflective vest detection algorithm based on improved YOLOX-S [J]. Electronic Measurement Technology, 2022). However, these methods do not fully consider the positional correlation between protective equipment and the parts of the body wearing the equipment. During the detection process, there may be situations where protective equipment is identified from the image to be detected, but the protective equipment is not worn by the person.

[0005] Therefore, this invention proposes a method for identifying protective gear worn by construction site inspection robots based on human posture detection. This method can not only simultaneously identify the wearing status of safety helmets, gloves, and reflective vests, but also extract key human body points related to protective gear wearing. By using these key human body points, it can locate whether protective gear such as safety helmets, gloves, and reflective vests are worn on the person, thereby improving the recognition accuracy. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the technical problem this invention aims to solve is to provide a method for recognizing protective clothing worn by building inspection robots based on human posture detection.

[0007] The technical solution adopted by the present invention to solve the aforementioned technical problem is as follows:

[0008] A method for recognizing protective clothing worn by building inspection robots based on human posture detection includes the following steps:

[0009] Step 1: Acquire color images, depth images, and highlight color images containing construction workers. The highlight color images are acquired under flash conditions.

[0010] The second step involves obtaining 25 key points of the human body through human posture recognition, selecting key points related to protective clothing, and transforming the coordinates of these key points from the depth camera coordinate system to the color camera coordinate system. Protective clothing recognition includes three types of protective equipment: gloves, helmets, and reflective vests.

[0011] The third step is to convert the coordinates of the key points related to the protective clothing in the color camera coordinate system into the image form by calculating the pixel value of each pixel in the human posture heat map, and generate the human posture heat map.

[0012] The fourth step is to construct a protective clothing recognition model, which includes a first-level neural network, a second-level neural network, and a connection module located between the two neural networks. The human posture heatmap, color image, and bright color image are input into the first-level neural network for feature extraction. The connection module is used to adjust the size and number of channels of the output features of the first-level neural network. The output features of the connection module are input into the second-level neural network for feature processing to achieve protective clothing recognition.

[0013] Step 5: Train the protective clothing recognition model and then port the trained model to the construction inspection robot. During robot inspections, the robot will intelligently identify whether construction workers are wearing gloves, reflective vests, and safety helmets. The protective clothing recognition model will output the categories, confidence scores, and bounding boxes for the three types of protective equipment: gloves, reflective vests, and safety helmets. The output results of the protective clothing recognition model will be corrected, including corrections to the bounding boxes for reflective vests and the confidence scores for safety helmets.

[0014] The steps to correct the reflective vest recognition frame are as follows:

[0015] Step E1: Convert the color image and the highlight color image into a color grayscale image and a highlight grayscale image, respectively. Subtract the pixel values ​​of the pixels at the same position in the highlight grayscale image from those in the color grayscale image to obtain the brightness difference image.

[0016] Step E2: Process the brightness difference image to obtain the area where the reflective clothing is located in the bright color image, i.e., the foreground image;

[0017] Step E3: Calculate all independent foreground blocks in the foreground image using the seed filling method, then calculate the edge points of each foreground block using the insect-following method, and obtain the border of each foreground block based on the edge points using the minimum bounding rectangle method; classify the foreground blocks into three categories according to their positional relationship with the reflective vest recognition frame: foreground blocks whose entire area is within the reflective vest recognition frame, foreground blocks whose partial area is within the reflective vest recognition frame, and foreground blocks whose entire area is outside the reflective vest recognition frame.

[0018] Step E4: Summarize all edge points of all foreground blocks within the reflective vest recognition frame in both partial and full regions, as well as all points on the four borders of the reflective vest recognition frame. Obtain the minimum bounding rectangle of all summarized points using the minimum bounding rectangle method, which is the corrected reflective vest recognition frame. Replace the reflective vest recognition frame output by the protective clothing recognition model with the corrected reflective vest recognition frame to obtain the final reflective vest recognition frame.

[0019] The steps for correcting the confidence level of the safety helmet are as follows:

[0020] Step F1: Assuming the number of pixels within the helmet recognition frame output by the protective clothing recognition model is n, and the number of pixels occupied by the helmet within the helmet recognition frame is m, then the projected area hs of the helmet within the helmet recognition frame in the world coordinate system is:

[0021] hs = ps × m

[0022] m = n × 75%

[0023] In the formula, ps is the projected area of ​​a single pixel in the area where the safety helmet is located in the color image in the world coordinate system;

[0024] Step F2: Set the projected area interval of the safety helmet to [hs1, hs2]. Assuming the original confidence value of the safety helmet is conf1, the corrected confidence value conf2 is expressed as:

[0025]

[0026] Then, the corrected value of the safety helmet confidence score is used to replace the safety helmet confidence score output by the protective clothing recognition model to obtain the final safety helmet confidence score.

[0027] Furthermore, the specific process of the third step is as follows: First, create a blank image with the same size as the color image for each key point. Use the coordinates of the key points in the color image as the initial coordinates (x0, y0) of each pixel in the blank image. Then, substitute the pixel coordinates (x, y) of each pixel in the blank image into equation (5) to calculate the pixel value f(x, y) of each pixel and obtain the human posture heatmap.

[0028]

[0029] In the formula, A represents the amplitude, σ x σ y These represent the variances of the horizontal and vertical axes, respectively.

[0030] Furthermore, the first-level neural network is obtained by improving the VGG19 neural network, specifically by deleting layers 17, 18, and 19 of the VGG19 neural network, adjusting layer 1 to have three convolutional branches, and layer 2 to have two convolutional branches. Each convolutional branch has a 3×3 kernel size and 64 kernels. Layers 3 through 16 of the VGG19 neural network remain unchanged. The inputs to the three convolutional branches of layer 1 are a human pose heatmap, a color image, and a highlighted color image, respectively. First, the seven human pose heatmaps are processed by… After one convolutional branch, feature map CA1 is obtained; the color image and the highlighted color image are respectively processed through two other convolutional branches to obtain feature maps CB1 and CB1'; then, feature map CA1 is processed through one of the convolutional branches of the second layer structure to obtain feature map CA2; feature maps CB1 and CB1' are concatenated by channels and then processed through another convolutional branch of the second layer structure to obtain feature map CB2; finally, feature maps CA2 and CB2 are concatenated by channels and input into the third layer structure. After processing by the remaining structural layers, the output features of the first-level neural network are obtained.

[0031] The connection module includes a cascaded first convolutional block, a second convolutional block, a convolution operation, and a downsampling operation. Each convolutional block includes three convolutional operation branches. The feature maps obtained from the three convolutional operation branches of each convolutional block are concatenated through channels to obtain the output features of each convolutional block. The first convolutional operation branch of the first convolutional block includes a convolutional layer with a kernel size of 3×3 and a number of 64 kernels. The second convolutional operation branch of the first convolutional block includes a convolutional layer with a kernel size of 1×1 and a number of 256 kernels, and a convolutional layer with a kernel size of 3×3 and a number of 128 kernels. The third convolution operation branch of the first convolutional block includes a max pooling layer and a convolutional layer with a kernel size of 1×1 and a number of 64 kernels; the first convolution operation branch of the second convolutional block includes a convolutional layer with a kernel size of 1×1 and a number of 32 kernels; the second convolution operation branch of the second convolutional block includes a convolutional layer with a kernel size of 1×1 and a number of 128 kernels and a convolutional layer with a kernel size of 3×3 and a number of 64 kernels; the third convolution operation branch of the second convolutional block includes a max pooling layer and a convolutional layer with a kernel size of 1×1 and a number of 32 kernels.

[0032] The second-level neural network was obtained by improving the YOLOv4 neural network.

[0033] Furthermore, key points related to protective clothing include seven key points: left hand, right hand, left shoulder, right shoulder, midpoint of the spine, head, and neck. The left and right hands are used to identify the left and right gloves, the left and right shoulders and the midpoint of the spine are used to identify the reflective vest, and the head and neck are used to identify the safety helmet.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] 1. This invention enables the simultaneous identification of safety helmets, gloves, and reflective vests. By detecting human posture, it extracts seven key points related to protective gear wearing, locates gloves, reflective vests, and safety helmets using these key points, and detects whether these protective gears are worn on the person. It fully considers the positional correlation between the protective gear and the parts of the body wearing the gear, avoids interference from similar objects in the image on the identification of the protective gear, and helps to improve the recognition accuracy.

[0036] 2. The protective clothing recognition model of this invention includes a first-level neural network, a second-level neural network, and a connection module between the two neural networks. The first-level neural network is an improvement on the VGG19 neural network, which fully extracts feature information from human posture heatmaps, color images, and highlighted color images. The second-level neural network is an improvement on the YOLOv4 neural network, which integrates, analyzes, and judges the feature information extracted by the first-level neural network, and then makes a recognition decision on the protective clothing, ensuring the reliability and accuracy of protective clothing recognition. Compared with the prior art, this invention fully utilizes human posture information, comprehensively processes multi-source information, and achieves a high degree of reliability in detection while diversifying the types of protective clothing.

[0037] 3. To improve the reliability of the protective clothing recognition model's output data, the confidence scores for reflective vest recognition boxes and safety helmets were corrected. For the reflective vest recognition box correction, light reflection information was fully utilized; the accuracy of the recognition box was improved by leveraging the feature differences between the color image and the highlighted color image. For the safety helmet confidence score correction, the score was adjusted based on the helmet's projected area in the world coordinate system and a reasonable range of projected areas, thus improving the robustness of the safety helmet confidence score.

[0038] 4. Using a construction worker training base or construction site in a higher vocational college as the test environment, 500 test images were collected. The test results show that the method of the present invention can achieve a recognition accuracy of 97.8% for safety helmets, 98.2% for reflective vests, and 95.2% for gloves. It can be seen that the method of the present invention can not only recognize safety helmets, reflective vests and gloves at the same time, but also has high recognition accuracy for all three types of protective equipment. Attached Figure Description

[0039] Figure 1 This is an overall flowchart of the method of the present invention;

[0040] Figure 2 It is a distribution map of 25 key points in the human body;

[0041] Figure 3 This is the architecture diagram of the first-level neural network;

[0042] Figure 4 This is the architecture diagram of the connection module;

[0043] Figure 5 This is the architecture diagram of the second-level neural network;

[0044] Figure 6 This is a visual diagram illustrating the identification results of the method of the present invention. Detailed Implementation

[0045] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments, but this does not limit the scope of protection of this application.

[0046] This invention relates to a method for recognizing protective clothing worn by building inspection robots based on human posture detection (hereinafter referred to as the method, see [link]). Figures 1-6 The process includes the following steps:

[0047] Step 1: Collect color images, depth images, and highlight color images containing construction workers;

[0048] The Kinect V2 camera was used to capture color and depth images of construction workers. The color images had a resolution of 1920×1080 pixels, and the depth images had a resolution of 512×424 pixels. The flash was then turned on, and the Kinect V2 camera was used to capture a color image of the flash, resulting in a highlight color image with the same resolution as the color images.

[0049] The second step involves extracting the coordinates of seven key points related to protective clothing in the depth camera coordinate system through human posture recognition, and then transforming the coordinates of the seven key points in the depth camera coordinate system to the color camera coordinate system through coordinate transformation. The seven key points are the left hand, right hand, left shoulder, right shoulder, midpoint of the spine, head, and neck. The left hand and right hand are used to identify the left and right gloves, the left shoulder, right shoulder, and midpoint of the spine are used to identify the reflective vest, and the head and neck are used to identify the safety helmet.

[0050] Human pose recognition was performed using depth images to identify key points corresponding to key parts of the human body. A total of 25 key points were obtained, and their coordinates in the depth camera coordinate system were obtained. The distribution of these 25 key points is shown below. Figure 2 As shown; the method for recognizing human poses can be found in the literature "Real-Time Human Pose Recognition in Parts from Single Depth Images" (Shotton J, Fitzgibbon A, Cook M, et al. Real-Time Human Pose Recognition in Parts from Single Depth Images[J]. Springer Berlin Heidelberg, 2013.).

[0051] Seven key points related to protective clothing were selected from the 25 key points, and the coordinates of the seven key points in the depth camera coordinate system were transformed to the color camera coordinate system according to Equation (1).

[0052] J r =Jd ×M d&r (1)

[0053] In the formula, This indicates the coordinates of the key points in the color camera coordinate system. M represents the coordinates of the keypoint in the depth camera coordinate system; d&r The coordinate transformation matrix is ​​expressed as follows:

[0054] M d&r =RT (2)

[0055] In the formula, R and T represent the extrinsic rotation matrix and extrinsic translation vector, respectively, and their expressions are as follows:

[0056]

[0057]

[0058] In the formula, R r T r Let R represent the rotation matrix and translation vector of the color camera, respectively. d T d Let represent the rotation matrix and translation vector of the depth camera, respectively;

[0059] The third step is to convert the seven key points from coordinate form into image form, generating seven human posture heatmaps.

[0060] To facilitate the processing of coordinate parameters by the protective clothing recognition model, the coordinates need to be further converted into an image format. First, a blank image of the same size as the color image is created for each key point, and the coordinates (x, y, z) of the key points in the color image are then mapped. r ,y r ) is used as the initial coordinates of each pixel in the blank image. Then, the pixel coordinates (x,y) of each pixel in the blank image are substituted into the two-dimensional Gaussian function of formula (5) to calculate the pixel value f(x,y) of each pixel. Human posture heatmaps are generated on the basis of the blank image. A total of 7 human posture heatmaps are generated.

[0061]

[0062] In the formula, A represents the amplitude, σ x σ y Let x and y represent the variances of the x and y coordinates, respectively, and let (x0, y0) represent the initial coordinates of the pixel. (x0, y0) = (x... r ,y r );

[0063] The fourth step is to construct a protective clothing recognition model, which includes a first-level neural network, a second-level neural network, and a connection module located between the two levels of neural networks.

[0064] The first-level neural network serves as a feature extraction network, used to extract features from various images. This first-level neural network is an improvement upon the VGG19 (Visual Geometry Group, 19-weight layers) neural network. Specifically, layers 17, 18, and 19 of the VGG19 neural network are removed. Layer 1 is adjusted to have three convolutional branches, and layer 2 to have two convolutional branches. Each convolutional branch (CONV3×3-64) has a 3×3 kernel size and 64 kernels. Layers 3 through 16 of the VGG19 neural network remain unchanged. See [link / reference]. Figure 3 The inputs to the three convolutional branches of the first layer are a human pose heatmap, a color image, and a highlighted color image, respectively. First, the weights of one convolutional branch of the first layer of the VGG19 neural network are randomly initialized using white noise with a variance of 0.1. After passing the seven human pose heatmaps through this convolutional branch, feature map CA1 is obtained. The other two convolutional branches of the first layer retain their original weights. After passing the color image and the highlighted color image through these two convolutional branches, feature maps CB1 and CB1' are obtained, respectively. Then, the second layer is randomly initialized. The weights of one of the convolutional branches are used to obtain feature map CA2 after feature map CA1 passes through this convolutional branch. The other convolutional branch of the second layer retains its original weights. Feature maps CB1 and CB1' are concatenated to 128 channels and then passed through this convolutional branch to obtain feature map CB2. Finally, feature maps CA2 and CB2 are concatenated to 128 channels and then input into the third layer. After processing by the remaining layers, a feature map P with a size of 244×244 and 512 channels is obtained, which is the output feature of the first-level neural network.

[0065] In order for the second-level neural network to process the output features of the first-level neural network, the size and number of channels of the output features of the first-level neural network are adjusted through the connection module; for example... Figure 4As shown, the connection module includes a cascaded first convolutional block, a second convolutional block, a convolution operation, and a downsampling operation. Each convolutional block includes three convolutional operation branches. The feature maps obtained from the three convolutional operation branches of each convolutional block are concatenated through channels to obtain the output features of each convolutional block. The first convolutional operation branch of the first convolutional block includes a convolutional layer with a kernel size of 3×3 and 64 kernels. The second convolutional operation branch of the first convolutional block includes a convolutional layer with a kernel size of 1×1 and 256 kernels, and a convolutional layer with a kernel size of 3×3 and 128 kernels. The third convolutional operation branch of the first convolutional block includes a max-pooling layer and a convolutional layer with a kernel size of 1×1 and 64 kernels. The first convolutional operation branch of the second convolutional block includes a convolutional layer with a kernel size of 1×1 and 32 kernels. The second convolutional operation branch of the second convolutional block includes a convolutional layer with a kernel size of 1×1 and 128 channels, and a convolutional layer with a kernel size of 3×3 and 64 channels. The third convolutional operation branch of the second convolutional block includes a max pooling layer and a convolutional layer with a kernel size of 1×1 and 32 channels. After the feature map P passes through the first convolutional block, it becomes a feature map M with a size of 244×244 and 256 channels, which is the output feature of the first convolutional block. After the feature map M passes through the second convolutional block, it becomes a feature map N with a size of 244×244 and 128 channels. After the feature map N passes through the convolutional operation and downsampling operation, it becomes a feature map Q with a size of 104×104 and 128 channels, which is the output feature of the connection module. The convolutional operation has a kernel size of 3×3, 128 channels, and a stride of 2.

[0066] The output features of the connection module are input into the second-level neural network. This second-level neural network, acting as a feature processing network, integrates, analyzes, and judges diverse protective information, including color images, human posture heatmaps, and high-brightness color images, to achieve protective clothing recognition. The second-level neural network is obtained by improving the YOLOv4 neural network; specifically, the DarknetConv2D_BN_Mish(416,416,32) and Resblock_body(208,208,64)×1 sub-modules of the YOLOv4 neural network are deleted, while the rest of the structure remains unchanged. Therefore, the input size of the second-level neural network is 104×104, and the number of channels is 128. Figure 5 As shown, the YOLOv4 neural network consists of three parts: a backbone network, a neck network, and a head network. The backbone network is an adjusted CSPDarknet53.

[0067] Step 5: Train the protective clothing recognition model and then port the trained model to the construction inspection robot. During robot inspections, the robot will intelligently identify whether construction workers are wearing gloves, reflective vests, and safety helmets. The protective clothing recognition model can identify these three types of protective equipment and output the category, confidence level, and bounding box. To further improve the accuracy of the output results, the output results of the protective clothing recognition model will be corrected, including the correction of the reflective vest bounding box and the safety helmet confidence level.

[0068] The steps to correct the reflective vest recognition frame are as follows:

[0069] Step E1: Convert the color image and the highlight color image acquired in the first step into a color grayscale image and a highlight grayscale image, respectively. Subtract the pixel values ​​of the pixels at the same position in the highlight grayscale image from those in the color grayscale image to obtain the brightness difference image.

[0070] Step E2: Perform median filtering on the brightness difference image to remove high-brightness noise; use OTSU (Great Law Method) segmentation to segment the median-filtered image, and then perform an opening operation (first erosion, then dilation) on the segmented image to obtain the area where the reflective clothing is located in the bright color image, i.e., the foreground image;

[0071] Step E3: Calculate all independent foreground blocks in the foreground image using the seed filling method, then calculate the edge points of each foreground block using the insect-following method, and obtain the border of each foreground block based on the edge points using the minimum bounding rectangle method; mark the foreground blocks according to their positional relationship with the reflective vest recognition frame, marking the foreground blocks whose entire area is within the reflective vest recognition frame as 0, marking the foreground blocks whose partial area is within the reflective vest recognition frame as 1, and marking the foreground blocks whose entire area is outside the reflective vest recognition frame as -1;

[0072] Step E4: Summarize all edge points of all foreground blocks within the reflective vest recognition frame in both partial and complete regions, as well as all points on the four borders of the reflective vest recognition frame. Obtain the minimum bounding rectangle of all summarized points using the minimum bounding rectangle method, which is the corrected reflective vest recognition frame. Replace the reflective vest recognition frame output by the protective clothing recognition model with the corrected reflective vest recognition frame to obtain the final reflective vest recognition frame.

[0073] The steps for correcting the confidence level of the safety helmet are as follows:

[0074] Step F1: Based on the distance between the helmet and the camera, and the camera model parameters, calculate the projected area ps of a single pixel in the helmet region of the color image in the world coordinate system. Assuming the number of pixels within the helmet recognition frame output by the protective clothing recognition model is n, and the number of pixels occupied by the helmet within the helmet recognition frame is m, then the projected area hs of the helmet within the helmet recognition frame in the world coordinate system is:

[0075] hs = ps × m

[0076] m = n × 75%

[0077] Step F2: Set the projected area interval of the safety helmet to [hs1, hs2]. Assuming the original confidence value of the safety helmet is conf1, the corrected confidence value conf2 is expressed as:

[0078]

[0079] Then, the corrected value of the safety helmet confidence score is used to replace the safety helmet confidence score output by the protective clothing recognition model to obtain the final safety helmet confidence score.

[0080] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1.A method for recognizing protective wear of a building inspection robot based on human posture detection, characterized in that, Includes the following steps: Step 1: Acquire color images, depth images, and highlight color images containing construction workers. The highlight color images are acquired under flash conditions. The second step involves obtaining 25 key points of the human body through human posture recognition, selecting key points related to protective clothing, and transforming the coordinates of these key points in the depth camera coordinate system to the color camera coordinate system. Protective clothing recognition includes three types of protective equipment: gloves, safety helmets, and reflective vests. The third step is to convert the coordinates of the key points related to the protective clothing in the color camera coordinate system into the image form by calculating the pixel value of each pixel in the human posture heat map, and generate the human posture heat map. The fourth step is to construct a protective clothing recognition model, which includes a first-level neural network, a second-level neural network, and a connection module located between the two neural networks. The human posture heatmap, color image, and bright color image are input into the first-level neural network for feature extraction. The connection module is used to adjust the size and number of channels of the output features of the first-level neural network. The output features of the connection module are input into the second-level neural network for feature processing to achieve protective clothing recognition. Step 5: Train the protective clothing recognition model and then port the trained model to the construction inspection robot. During robot inspections, the robot will intelligently identify whether construction workers are wearing gloves, reflective vests, and safety helmets. The protective clothing recognition model will output the categories, confidence scores, and bounding boxes for the three types of protective equipment: gloves, reflective vests, and safety helmets. The output results of the protective clothing recognition model will be corrected, including corrections to the bounding boxes for reflective vests and the confidence scores for safety helmets. The steps to correct the reflective vest recognition frame are as follows: Step E1: Convert the color image and the highlight color image into a color grayscale image and a highlight grayscale image, respectively. Subtract the pixel values ​​of the pixels at the same position in the highlight grayscale image from those in the color grayscale image to obtain the brightness difference image. Step E2: Process the brightness difference image to obtain the area where the reflective clothing is located in the bright color image, i.e., the foreground image; Step E3: Calculate all independent foreground blocks in the foreground image using the seed filling method, then calculate the edge points of each foreground block using the insect-following method, and obtain the border of each foreground block based on the edge points using the minimum bounding rectangle method; classify the foreground blocks into three categories according to their positional relationship with the reflective vest recognition frame: foreground blocks whose entire area is within the reflective vest recognition frame, foreground blocks whose partial area is within the reflective vest recognition frame, and foreground blocks whose entire area is outside the reflective vest recognition frame. Step E4: Summarize all edge points of all foreground blocks within the reflective vest recognition frame in both partial and full regions, as well as all points on the four borders of the reflective vest recognition frame. Obtain the minimum bounding rectangle of all summarized points using the minimum bounding rectangle method, which is the corrected reflective vest recognition frame. Replace the reflective vest recognition frame output by the protective clothing recognition model with the corrected reflective vest recognition frame to obtain the final reflective vest recognition frame. The steps for correcting the confidence level of the safety helmet are as follows: Step F1: Assume the number of pixels within the helmet recognition frame output by the protective clothing recognition model is . n The number of pixels occupied by the helmet within the helmet recognition frame is [number]. The projected area of ​​the safety helmet within the safety helmet recognition frame in the world coordinate system is then... for: ， ， In the formula, is the projection area of a single pixel point in the world coordinate system in the region where the safety helmet is located in the color image. Step F2: Set the projected area range of the safety helmet as follows Assuming the original confidence level of the safety helmet is 100%. Then the corrected value for the confidence level of the safety helmet Represented as: ， Then, the corrected value of the safety helmet confidence score is used to replace the safety helmet confidence score output by the protective clothing recognition model to obtain the final safety helmet confidence score. 2.The method of claim 1, wherein, The third step is as follows: First, create a blank image of the same size as the color image for each key point, and use the coordinates of the key points in the color image as the initial coordinates of each pixel in the blank image. Then, the pixel coordinates of each pixel in the blank image are... Substitute into equation (5) to calculate the pixel value of each pixel. , thus obtaining a human posture heatmap; （5） wherein denotes the amplitude, , denotes the variance of the horizontal, respectively vertical coordinate. 3.The method of claim 1, wherein, The first-level neural network is an improvement upon the VGG19 neural network. Specifically, layers 17, 18, and 19 of the VGG19 neural network are removed, and layer 1 is adjusted to have three convolutional branches, while layer 2 is adjusted to have two convolutional branches. Each convolutional branch has a 3×3 kernel size and 64 kernels. Layers 3 through 16 of the VGG19 neural network remain unchanged. The inputs to the three convolutional branches of layer 1 are a human pose heatmap, a color image, and a highlighted color image, respectively. First, seven human pose heatmaps are processed through one of the... After the convolutional branch, feature map CA1 is obtained; the color image and the highlighted color image are respectively processed through two other convolutional branches to obtain feature maps CB1 and CB1'; then, feature map CA1 is processed through one of the convolutional branches of the second layer structure to obtain feature map CA2; feature maps CB1 and CB1' are concatenated by channels and then processed through another convolutional branch of the second layer structure to obtain feature map CB2; finally, feature maps CA2 and CB2 are concatenated by channels and then input into the third layer structure. After processing by the remaining structural layers, the output features of the first-level neural network are obtained. The connection module includes a cascaded first convolutional block, a second convolutional block, a convolution operation, and a downsampling operation. Each convolutional block includes three convolutional operation branches. The feature maps obtained from the three convolutional operation branches of each convolutional block are concatenated through channels to obtain the output features of each convolutional block. The first convolutional operation branch of the first convolutional block includes a convolutional layer with a kernel size of 3×3 and a number of 64 kernels. The second convolutional operation branch of the first convolutional block includes a convolutional layer with a kernel size of 1×1 and a number of 256 kernels, and a convolutional layer with a kernel size of 3×3 and a number of 128 kernels. The third convolution operation branch of the first convolutional block includes a max pooling layer and a convolutional layer with a kernel size of 1×1 and a number of 64 kernels; the first convolution operation branch of the second convolutional block includes a convolutional layer with a kernel size of 1×1 and a number of 32 kernels; the second convolution operation branch of the second convolutional block includes a convolutional layer with a kernel size of 1×1 and a number of 128 kernels and a convolutional layer with a kernel size of 3×3 and a number of 64 kernels; the third convolution operation branch of the second convolutional block includes a max pooling layer and a convolutional layer with a kernel size of 1×1 and a number of 32 kernels. The second-level neural network is obtained by improving the YOLOv4 neural network. Specifically, the DarknetConv2D_BN_Mish(416,416,32) submodule and the Resblock_body(208,208,64)×1 submodule of the YOLOv4 neural network are deleted, while the rest of the structure remains unchanged. Therefore, the input size of the second-level neural network is 104×104 and the number of channels is 128. The YOLOv4 neural network consists of three parts: the backbone network, the neck network, and the head network. The backbone network is the adjusted CSPDarknet53. 4.The method of claim 1, wherein, Key points related to protective clothing include seven key points: left hand, right hand, left shoulder, right shoulder, midpoint of the spine, head, and neck. The left and right hands are used to identify the left and right gloves, the left and right shoulders and the midpoint of the spine are used to identify the reflective vest, and the head and neck are used to identify the safety helmet.

Images