A Secure Wearable Detection Method Based on Yolov7 and Deepsort

Abstract

This invention relates to a safety wear detection method based on YOLOv7 and Deepsort. It applies a person detection model to analyze captured images, obtaining person detection results. Then, the Deepsort network is used to achieve continuous person tracking between captured frames. Finally, the safety wear detection model is applied to perform further safety wear detection for continuous person tracking. An improved YOLOv7 network structure based on the LSAM attention mechanism enhances the network's ability to perceive small targets, improving detection accuracy. Furthermore, by adding velocity direction factors to the cost matrix during person tracking using the Deepsort network, the accuracy and robustness of target tracking are improved. This allows for better capture of key features in the image, more effective target tracking, and significantly reduced false positives and false negatives in the safety wear detection process.

Description

Technical Field

[0001] This invention relates to a secure wearable detection method based on YOLOv7 and Deepsort, belonging to the field of computer vision and deep learning technology. Background Technology

[0002] In recent years, intelligent monitoring technology for industrial sites based on video surveillance has attracted widespread attention. Intelligent video surveillance utilizes computer vision technology to detect, track, identify, and understand targets of interest in a scene, and proactively alarms and records abnormal events, effectively assisting workers in work supervision. Currently, intelligent video surveillance technology is mainly applied in industrial production sites, including product quality monitoring, production equipment fault monitoring, and safety monitoring of industrial workers, with worker safety monitoring being particularly important. In industrial site operations, safety helmets and protective clothing are essential equipment for worker safety, playing a crucial role in protecting workers' lives. Therefore, workers must wear protective clothing and safety helmets correctly when entering the work site. However, in actual work, due to insufficient attention to safety production, workers may not wear uniforms and safety helmets correctly according to relevant regulations, creating potential safety hazards.

[0003] An analysis and summary of the current state of research on wearable device testing both domestically and internationally reveals that a significant amount of research has been conducted on safety wearable device testing, yielding certain results. However, some problems remain, such as: sensor-based detection methods having low applicability and susceptibility to electromagnetic interference; and traditional image detection methods having low robustness and relying on manually designed features.

[0004] In addition, deep learning-based detection methods have the following problems: some detection models prioritize detection speed at the expense of detection accuracy; some detection models have high detection accuracy but slow detection speed, making it impossible to monitor construction workers in real time. Therefore, in enterprise construction safety management, it is impossible to simultaneously meet the requirements of fast detection speed and high accuracy of the model. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a safety wear detection method based on YOLOv7 and Deepsort, which enhances the perception of small targets, improves detection accuracy, and greatly reduces the false detection and false negative rates of worker safety wear detection through innovative tracking design.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention designs a safety wear detection method based on YOLOv7 and Deepsort, which performs the following steps A to F in real time on the image captured by the image capture device to realize real-time safety wear detection of the person being captured;

[0007] Step A. Based on the pre-trained improved Yolov7 network structure for person detection, which takes the image as input and outputs the person detection results in the image, analyze the currently captured image to obtain the person detection results in the currently captured image. The person detection results include whether there are people in the image and the location of the people in the image. Then proceed to Step B.

[0008] Step B. Determine whether a person exists in the current image. If so, obtain the local image of the current person in each frame of the current image and proceed to step C; otherwise, the processing of the current image ends.

[0009] Step C. Determine whether there is a previous frame image that is adjacent to the current captured image and contains a partial image of the person in the historical time direction. If yes, proceed to step D; otherwise, for each frame of the current partial image of the person in the current captured image, mark each frame with a tracking mark that is different from the already marked tracking mark and is different from each other. This constitutes a new tracking mark corresponding to each frame of the current partial image of the person in the current captured image, and proceed to step F.

[0010] Step D. Based on the tracking markers corresponding to the local images of each person in the previous frame, and considering the historical time direction, according to the consecutive local images of each person corresponding to each tracking marker, apply the preset feature extraction network and the Deepsort network to track and detect the local images of the current person in each frame of the current frame.

[0011] Specifically, for each frame of the current person's local image that has been successfully tracked, the corresponding current person's local image is marked with each tracking mark that has been successfully tracked, forming the old tracking mark corresponding to each frame of the current person's local image, and then proceeding to step E;

[0012] For each frame of the current local image of the person that was not successfully tracked, a new tracking mark is marked that is different from the existing tracking mark and is different from each other, thus forming a new tracking mark corresponding to each frame of the current local image of the person, and then proceed to step F;

[0013] Step E. Based on the current person's local images in each frame corresponding to the old tracking markers, for each tracking marker, based on the consecutive person's local images corresponding to the tracking marker, if the person corresponding to the tracking marker is in the correct safety wearing state, then no processing is performed on the current person's local image corresponding to the tracking marker;

[0014] If the person corresponding to the tracking tag is wearing the wrong safety gear, an alarm will be triggered on the local image of the person corresponding to the tracking tag.

[0015] If the person corresponding to the tracking tag is in a temporary analysis wearing state, then the following processing shall be performed:

[0016] First, a pre-trained safety wear detection model with an improved Yolov7 network structure is applied. This model takes a local image of a person as input and outputs the result of whether the person in the local image is wearing various preset types of safety clothing. The model is then used to analyze the current local image of the person corresponding to the tracking marker to obtain the result of whether the person in the current local image is wearing various preset types of safety clothing.

[0017] Then, it is determined whether all types of safety clothing are worn under the cumulative analysis of the safety wear detection model for the consecutive local images of the person corresponding to the tracking marker. If so, the person corresponding to the tracking marker is defined as being in a correct safety wear state, and no processing is performed on the local image of the current person corresponding to the tracking marker. Otherwise, it is further determined whether the number of frames of the consecutive local images of the person corresponding to the tracking marker has reached the preset safety detection frame limit. If so, the person corresponding to the tracking marker is defined as being in an incorrect safety wear state, and an alarm is triggered on the local image of the current person corresponding to the tracking marker. Otherwise, the person corresponding to the tracking marker is kept in a temporary analysis and wear state, and no processing is performed on the local image of the current person corresponding to the tracking marker.

[0018] Step F. For each frame of the current person's local image corresponding to the new tracking marker, first apply a pre-trained safety wear detection model with an improved Yolov7 network structure that takes the person's local image as input and outputs the result of whether the person is wearing the preset types of safety clothing in the local image. Analyze the current person's local image to obtain the result of whether the person is wearing the preset types of safety clothing in the current person's local image. Then determine whether the person in the local image has worn all the preset types of safety clothing. If so, define the person corresponding to the tracking marker as being in a correct safety wear state, and do not process the current person's local image corresponding to the tracking marker. Otherwise, define the person corresponding to the tracking marker as being in a temporary safety wear state, and do not process the current person's local image corresponding to the tracking marker.

[0019] As a preferred embodiment of the present invention: the improved YOLOv7 network structure includes a backbone network and a head network. The backbone network includes, in the direction of data flow, a CBS layer, a CBS layer, a CBS layer, an ELAN layer, an LSAM layer, an MPconv layer, an ELAN layer, an LSAM layer, an MPconv layer, an ELAN layer, an LSAM layer, an MPconv layer, an ELAN layer, an LSAM layer, an MPconv layer, an ELAN layer, an LSAM layer, an MPconv layer, an ELAN layer, an LSAM layer, and an SPP layer connected in series. The input end of the first CBS layer constitutes the input end of the improved YOLOv7 network structure. The output end of the first LSAM layer constitutes the first output end of the backbone network. The output end of the second LSAM layer constitutes the second output end of the backbone network. The output end of the third LSAM layer constitutes the third output end of the backbone network. The output end of the SPP layer constitutes the fourth output end of the backbone network.

[0020] The detection head network head includes four front-end CBS layers, three first combination layers, three second combination layers, and four output layers. Each output of the backbone network is connected to one of the four front-end CBS layers in the detection head network head. The structures of each first combination layer are identical. Each first combination layer, from its input to its output, includes a CBS layer, an Unsample layer, a Concat layer, and an ELAN-W layer connected in series. The input of the CBS layer in each first combination constitutes the input of that first combination, and the output of the ELAN-W layer constitutes the output of that first combination. The first combination layers are connected in series in the direction of data flow. The input of the Concat layer in each first combination constitutes the intermediate input of that first combination. Each second combination layer... The structures are identical to each other. Each second combined layer includes, from its input to its output, a series of MPconv, Concat, and ELAN-W layers connected in series. The input of the MPconv layer in each second combined layer constitutes the input of the second combined layer, and the output of the ELAN-W layer in each second combined layer constitutes the output of the second combined layer. The second combined layers are connected in series in the direction of data flow. The input of the Concat layer in each second combined layer constitutes the intermediate input of each second combined layer. The structures of the output layers are identical to each other. Each output layer includes, from its input to its output, a series of REPconv and Conv layers connected in series. The input of the REPconv layer in each output layer constitutes the input of the output layer, and the output of the Conv layer in each output layer constitutes the output of the output layer.

[0021] The output of the front-end CBS layer connected to the first output of the backbone network is connected to the intermediate input of the third first combination layer in sequence. The output of the front-end CBS layer connected to the second output of the backbone network is connected to the intermediate input of the second first combination layer in sequence. The output of the front-end CBS layer connected to the third output of the backbone network is connected to the intermediate input of the first first combination layer in sequence. The output of the front-end CBS layer connected to the fourth output of the backbone network is connected to the input of the first first combination layer in sequence. The output of the third first combination layer in sequence is connected to the input of the first second combination layer in sequence. The output of the second first combination layer in sequence is connected to the intermediate input of the first second combination layer in sequence. The output of the first first combination layer in sequence is connected to the intermediate input of the second second combination layer in sequence. The output of the front-end CBS layer connected to the fourth output of the backbone network is also connected to the intermediate input of the third second combination layer in sequence. The outputs of the third first combination layer, the first second combination layer, the second second combination layer, and the third second combination layer in sequence are connected one-to-one to the inputs of each output layer. The outputs of each output layer constitute the outputs of the improved Yolov7 network structure.

[0022] As a preferred embodiment of the present invention: the LSAM layers have the same structure, and each LSAM layer includes a CAM module, a SAM module, a first element-wise summation module, an activation function module, a normalization module, a matrix multiplication module, and a second element-wise summation module. In the structure of each LSAM layer, the input terminals of the CAM module and the SAM module constitute the two input terminals of the LSAM layer, which are used to receive feature maps. The CAM module is used to calculate the channel attention weight matrix E of the received feature map. c The output, the SAM module, is used to calculate the spatial attention weight matrix E of the received feature map. s The outputs of the CAM module and the SAM module are connected to the input of the first element-wise summation module. The output of the first element-wise summation module is then connected to the input of the normalization module via the activation function module. The output of the normalization module outputs the spatial and channel attention weight matrix E. cs The output of the activation module is connected to the input of the matrix multiplication module. The matrix multiplication module performs the multiplication on the received E. cs The first module performs matrix multiplication on the feature map to obtain the processing result. The output of the matrix multiplication module is connected to the input of the second element-wise summation module. The second element-wise summation module performs element-wise summation on the feature map based on the processing result of the matrix multiplication module, thereby updating the feature map output.

[0023] As a preferred technical solution of the present invention: the CBS layers have the same structure, and each CBS layer includes a Conv layer, a BN layer and a SiLU layer connected in series from its input end to its output end;

[0024] Each MPconv layer has the same structure. In each MPconv layer structure, the input end of the MPconv layer is connected to two parallel structures. One structure is connected in series with the MaxPool layer and the CBS layer, and the other structure is connected in series with two CBS layers. The output end of the two structures is connected to the input end of the Concat layer, and the output end of the Concat layer constitutes the output end of the MPconv layer.

[0025] Each ELAN layer has the same structure. In each ELAN layer structure, the input terminal of the ELAN layer is connected to two parallel structures. The first structure is five CBS layers connected in series. The second structure is a single CBS layer. The output terminals of the first, third, and fifth CBS layers in the first structure and the CBS layer output terminal in the second structure are connected to the input terminal of the Concat layer. The output terminal of the Concat layer is connected to the input terminal of another CBS layer. The output terminal of this CBS layer constitutes the output terminal of the ELAN layer.

[0026] Each ELAN-W layer has the same structure. In each ELAN-W layer structure, the input of the ELAN-W layer is connected to two parallel structures. The first structure is five CBS layers connected in series in sequence. The second structure is a single CBS layer. The outputs of each CBS layer in the first structure and the outputs of the CBS layer in the second structure are connected to the input of the Concat layer. The output of the Concat layer is connected to the input of another CBS layer. The output of this CBS layer constitutes the output of the ELAN-W layer.

[0027] Each REPconv layer has the same structure. In each REPconv layer structure, the input end of the REPconv layer is connected to three identical parallel structures. Each structure includes a series Conv layer and a BN layer. The output end of each structure is connected to the input end of the Add layer. The output end of the Add layer is connected to the input end of the SiLU layer. The output end of the SiLU layer constitutes the output end of the REPconv layer.

[0028] In the SPP layer structure, the input of the SPP layer is connected to the input of the first CBS layer. The output of the first CBS layer is connected to the input of the second CBS layer, the input of the Concat layer, and the input of three parallel MaxPool layers. The input of the Concat layer is also connected to the output of the three parallel MaxPool layers. The output of the Concat layer is connected to the input of the third CBS layer. The outputs of the second and third CBS layers are connected to the input of another Concat layer. The output of the other Concat layer constitutes the output of the SPP layer.

[0029] As a preferred technical solution of the present invention: Regarding the person detection model, based on a preset number of sample images of each frame with known positions of each person detection box in the image, the sample images are used as input and the positions of the person detection boxes in the sample images are used as output. The model is trained on an improved Yolov7 network structure to obtain the person detection model.

[0030] Regarding the safety wear detection model, it is based on a preset number of sample images of people in each frame, each containing a known number of frames showing whether a person is wearing a preset type of safety clothing. The model is trained on an improved Yolov7 network structure, with the sample images as input and the output showing whether a person is wearing a preset type of safety clothing.

[0031] As a preferred technical solution of the present invention: for each frame of sample image, rotation, horizontal adjustment, cropping, and adjustment of brightness, contrast, saturation, and hue are performed separately to obtain new sample images obtained by each execution, thereby expanding each frame of sample images. Then, based on the expanded frame of sample images, the improved Yolov7 network structure is trained to obtain a person detection model.

[0032] For each frame of the human sample image, new human sample images are obtained by performing rotation, horizontal adjustment, cropping, and adjusting the brightness, contrast, saturation, and hue of the image. This expands each frame of the human sample image. Then, based on the expanded human sample images, the improved Yolov7 network structure is trained to obtain a safety wear detection model.

[0033] As a preferred technical solution of the present invention: In step D, for the tracking marker i corresponding to each person's local image in the previous frame and the current person's local image j in each frame of the current image, the following steps D1 to D4 are performed to detect whether the person's local image corresponding to the tracking marker i in the previous frame and the current person's local image j are tracked, thereby realizing the tracking detection of the current person's local images in each frame of the current image.

[0034] Step D1. First, apply the Kalman filter in the Deepsort network to obtain the predicted bounding box of the local image of the person corresponding to the tracking marker i in the previous frame in the current frame. Then, apply the pre-trained feature extraction model for extracting preset features in the image to extract the preset feature vectors y of the predicted bounding box of the local image of the person. i Simultaneously, extract the preset feature vectors d of the detection box of the current local image j of the person. j Finally, use the following formula:

[0035]

[0036] Obtain the Mahalanobis distance motion feature d between the predicted bounding box of the local image of the person corresponding to the tracking marker i and the detection box of the current local image j of the person. (1) (i, j), and proceed to step D2; where Q i The covariance matrix represents the relationship between the predicted bounding box of the local image of the person corresponding to tracking marker i and the detection bounding box of the current local image of the person j;

[0037] Step D2. Based on the appearance description features corresponding to the local images of each person in succession corresponding to the tracking marker i. And the appearance description features r of the detection box in the current local image j of the person. j According to the following formula:

[0038]

[0039] Obtain the minimum cosine value d of the appearance description features between each consecutive local image of the person corresponding to the tracking marker i and the current local image j. (2) (i, j), then proceed to step D3; where, This indicates the nth local image of each person corresponding to the tracking marker i. i The appearance description features corresponding to a partial image of a person, (r j ) T Indicates r j transpose;

[0040] Step D3. Based on the preset weight λ, use the following formula:

[0041] G(i,j)=λd (1) (i,j)+(1-λ)d (2) (i, j)

[0042] Get d (1) (i, j) and d (2) The association cost G(i,j) between (i,j) is calculated, and then proceed to step D4;

[0043] Step D4. According to the Hungarian algorithm, use the following formula:

[0044] G′(i,j)=G(i,j)+βG(i,j,v)

[0045] The association cost G′(i,j) between the local image of the person corresponding to the tracking marker i in the previous frame and the current local image j of the person is obtained. Combined with the preset association cost threshold, it is determined whether there is an association tracking between the local image of the person corresponding to the tracking marker i in the previous frame and the current local image j of the person. Here, β represents the preset weighting factor, v represents the trajectory direction of the local image of the person corresponding to the tracking marker i in the previous frame, and G(i,j,v) represents the direction difference between the trajectory direction formed by connecting the local image of the person corresponding to the tracking marker i in the previous frame and the current local image j of the person and v. This direction difference is represented by the absolute difference of the radian angle.

[0046] The safety wearable detection method based on Yolov7 and Deepsort described in this invention has the following technical advantages compared with existing technologies:

[0047] (1) The present invention designs a safety wear detection method based on Yolov7 and Deepsort, introduces the LSAM attention mechanism, innovatively designs an improved Yolov7 network structure, trains a person detection model and a safety wear detection model, applies the person detection model to analyze the captured images to obtain the person detection results, then applies the Deepsort network to realize continuous person tracking between each frame of captured images, and finally applies the safety wear detection model to perform further safety wear detection for continuous person tracking; the improved network structure in the design can enhance the network's perception ability of small targets and improve detection accuracy, and in the process of applying the Deepsort network for person tracking, the velocity direction factor is added to the cost matrix, which improves the accuracy and robustness of target tracking, can better capture key features in the image, and more effectively track targets, greatly reducing the false detection and false negative rates in the safety wear detection process. Attached Figure Description

[0048] Figure 1This is a schematic diagram of the improved Yolov7 network structure designed in this invention;

[0049] Figure 2 This is a schematic diagram of the LSAM layer in the improved Yolov7 network structure designed in this invention;

[0050] Figure 3 This is a schematic diagram of the structure of other layers in the improved Yolov7 network structure designed in this invention. Detailed Implementation

[0051] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0052] The present invention designs a security wear detection method based on YOLOv7 and Deepsort, which performs the following steps A to F in real time on the image captured by the image capture device to realize real-time security wear detection of the person being captured.

[0053] Step A. Based on the pre-trained improved Yolov7 network structure for person detection, which takes the image as input and outputs the person detection results in the image, analyze the currently captured image to obtain the person detection results in the currently captured image. The person detection results include whether there are people in the image and the location of the people in the image. Then proceed to Step B.

[0054] In practical applications, the person detection model is based on a preset number of sample images of each frame with known positions of person detection boxes in the image. For each sample image, rotation, leveling, cropping, and adjustment of brightness, contrast, saturation, and hue are performed separately to obtain new sample images for each execution, thereby expanding each frame of sample images. Then, based on the expanded sample images, the model is trained on an improved Yolov7 network structure with the sample images as input and the positions of person detection boxes in the sample images as output to obtain the person detection model.

[0055] Step B. Determine whether a person exists in the current captured image. If so, obtain the local image of the current person in each frame of the current captured image and proceed to step C; otherwise, the processing of the current captured image ends.

[0056] Step C. Determine whether there is a previous frame image that is adjacent to the current captured image and contains a partial image of the person in the historical time direction. If yes, proceed to step D; otherwise, for each frame of the current partial image of the person in the current captured image, mark each frame with a tracking mark that is different from the already marked tracking mark and is different from each other. This constitutes a new tracking mark corresponding to each frame of the current partial image of the person in the current captured image, and proceed to step F.

[0057] Step D. Based on the tracking markers corresponding to the local images of each person in the previous frame, and considering the historical time direction, according to the consecutive local images of each person corresponding to each tracking marker, apply a preset feature extraction network and a Deepsort network to track and detect the local images of the current person in each frame of the current image.

[0058] In practical applications, the above tracking detection is specifically designed to detect whether the tracking marker i in the previous frame of the captured image and the current person's local image j in each frame of the captured image are tracked, according to steps D1 to D4 below. This allows for the detection of whether the tracking marker i in the previous frame of the captured image is tracked by the current person's local image j, thereby enabling the tracking detection of the current person's local images in each frame of the captured image.

[0059] Step D1. First, apply the Kalman filter in the Deepsort network to obtain the predicted bounding box of the local image of the person corresponding to the tracking marker i in the previous frame in the current frame. Then, apply the pre-trained feature extraction model for extracting preset features in the image to extract the preset feature vectors y of the predicted bounding box of the local image of the person. i Simultaneously, extract the preset feature vectors d of the detection box of the current local image j of the person. j Finally, use the following formula:

[0060]

[0061] Obtain the Mahalanobis distance motion feature d between the predicted bounding box of the local image of the person corresponding to the tracking marker i and the detection box of the current local image j of the person. (1) (i, j), and proceed to step D2; where Q i This represents the covariance matrix between the predicted bounding box of the local image of the person corresponding to tracking marker i and the detection bounding box of the current local image of the person j.

[0062] Step D2. Based on the appearance description features corresponding to the local images of each person in succession corresponding to the tracking marker i. And the appearance description features r of the detection box in the current local image j of the person. j According to the following formula:

[0063]

[0064] Obtain the minimum cosine value d of the appearance description features between each consecutive local image of the person corresponding to the tracking marker i and the current local image j. (2) (i, j), then proceed to step D3; where, This indicates the nth local image of each person corresponding to the tracking marker i.i The appearance description features corresponding to a partial image of a person, (r j ) T Indicates r j The transpose of .

[0065] Step D3. Based on the preset weight λ, use the following formula:

[0066] G(i,j)=λd (1) (i,j)+(1-λ)d (2) (i, j)

[0067] Get d (1) (i, j) and d (2) The association cost G(i,j) between (i,j) is calculated, and then the process proceeds to step D4.

[0068] Step D4. According to the Hungarian algorithm, use the following formula:

[0069] G′(i,j)=G(i,j)+βG(i,j,v)

[0070] The association cost G′(i,j) between the local image of the person corresponding to the tracking marker i in the previous frame and the current local image j of the person is obtained. Combined with the preset association cost threshold, it is determined whether there is an association tracking between the local image of the person corresponding to the tracking marker i in the previous frame and the current local image j of the person. Here, β represents the preset weighting factor, v represents the trajectory direction of the local image of the person corresponding to the tracking marker i in the previous frame, and G(i,j,v) represents the direction difference between the trajectory direction formed by connecting the local image of the person corresponding to the tracking marker i in the previous frame and the current local image j of the person and v. This direction difference is represented by the absolute difference of the radian angle.

[0071] Based on the above tracking and detection operation, for each frame of the current person's local image that has been successfully tracked, the corresponding current person's local image is marked with each tracking mark that has been successfully tracked, forming the old tracking mark corresponding to each frame of the current person's local image, and then proceeding to step E;

[0072] For each frame of the current local image of the person that was not successfully tracked, a new tracking mark is marked that is different from the existing tracking mark and is different from each other, thus forming a new tracking mark corresponding to each frame of the current local image of the person, and then proceeding to step F.

[0073] Step E. Based on the current character's local images in each frame corresponding to the old tracking markers, and for each tracking marker, analyze the consecutive character local images corresponding to that tracking marker as follows:

[0074] If the person corresponding to the tracking tag is wearing the correct safety gear, then no processing will be performed on the local image of the person corresponding to the tracking tag.

[0075] If the person corresponding to the tracking tag is wearing the wrong safety gear, an alarm will be triggered on the local image of the person corresponding to the tracking tag.

[0076] If the person corresponding to the tracking tag is in a temporary analysis of their attire status, then the following further processing is performed:

[0077] First, a pre-trained safety wear detection model with an improved Yolov7 network structure is applied. This model takes a local image of a person as input and outputs the result of whether the person is wearing various types of safety clothing in the local image. The model analyzes the local image of the person corresponding to the tracking marker to obtain the result of whether the person is wearing various types of safety clothing in the local image. In practical applications, the various types of safety clothing include safety helmets, work clothes, and work pants, and the model detects whether these three types of clothing are being worn.

[0078] Then, it is determined whether all types of safety clothing are worn under the cumulative analysis of the safety wear detection model for the continuous local images of the person corresponding to the tracking marker. If so, the person corresponding to the tracking marker is defined as being in a correct safety wear state, and no processing is performed on the local image of the current person corresponding to the tracking marker. Otherwise, it is further determined whether the number of frames of the continuous local images of the person corresponding to the tracking marker has reached the preset safety detection frame limit. If so, the person corresponding to the tracking marker is defined as being in an incorrect safety wear state, and an alarm is triggered on the local image of the current person corresponding to the tracking marker. Otherwise, the person corresponding to the tracking marker is kept in a temporary analysis and wear state, and no processing is performed on the local image of the current person corresponding to the tracking marker.

[0079] Step F. For each frame of the current person's local image corresponding to the new tracking marker, first apply a pre-trained safety wear detection model with an improved Yolov7 network structure that takes the person's local image as input and outputs the result of whether the person is wearing the preset types of safety clothing in the local image. Analyze the current person's local image to obtain the result of whether the person is wearing the preset types of safety clothing in the current person's local image. Then determine whether the person in the local image has worn all the preset types of safety clothing. If so, define the person corresponding to the tracking marker as being in a correct safety wear state, and do not process the current person's local image corresponding to the tracking marker. Otherwise, define the person corresponding to the tracking marker as being in a temporary safety wear state, and do not process the current person's local image corresponding to the tracking marker.

[0080] In practical applications, the safety wear detection model is based on a preset number of sample images of people, each frame containing a known number of frames showing whether a person is wearing a preset type of safety clothing. For each frame of the sample images, rotation, leveling, cropping, and adjustments to brightness, contrast, saturation, and hue are performed to obtain new sample images, thus expanding the sample images. Then, based on these expanded sample images, using the sample images as input and the information on whether the person is wearing a preset type of safety clothing as output, the model is trained on an improved Yolov7 network structure to obtain the safety wear detection model.

[0081] The person detection model and the safety wear detection model here are both designed and trained based on an improved Yolov7 network structure, such as... Figure 1 As shown, the improved Yolov7 network structure includes a backbone network and a head network. In the specific design, the backbone network includes, in the direction of data flow, CBS layer, CBS layer, CBS layer, ELAN layer, LSAM layer, MPconv layer, ELAN layer, LSAM layer, MPconv layer, ELAN layer, LSAM layer, MPconv layer, ELAN layer, LSAM layer, MPconv layer, ELAN layer, LSAM layer, MPconv layer, ELAN layer, LSAM layer, and SPP layer connected in series. The input of the first CBS layer constitutes the input of the improved Yolov7 network structure, the output of the first LSAM layer constitutes the first output of the backbone network, the output of the second LSAM layer constitutes the second output of the backbone network, the output of the third LSAM layer constitutes the third output of the backbone network, and the output of the SPP layer constitutes the fourth output of the backbone network.

[0082] The detection head network head includes four front-end CBS layers, three first combination layers, three second combination layers, and four output layers. Each output of the backbone network is connected to one of the four front-end CBS layers in the detection head network head. The structures of each first combination layer are identical. Each first combination layer, from its input to its output, includes a CBS layer, an Unsample layer, a Concat layer, and an ELAN-W layer connected in series. The input of the CBS layer in each first combination constitutes the input of that first combination, and the output of the ELAN-W layer constitutes the output of that first combination. The first combination layers are connected in series in the direction of data flow. The input of the Concat layer in each first combination constitutes the intermediate input of that first combination. Each second combination layer... The structures are identical to each other. Each second combined layer includes, from its input to its output, a series of MPconv, Concat, and ELAN-W layers connected in series. The input of the MPconv layer in the second combined layer constitutes the input of the second combined layer, and the output of the ELAN-W layer in the second combined layer constitutes the output of the second combined layer. The second combined layers are connected in series in the direction of data flow. The input of the Concat layer in each second combined layer constitutes the intermediate input of each second combined layer. The structures of the output layers are identical to each other. Each output layer includes, from its input to its output, a series of REPconv and Conv layers connected in series. The input of the REPconv layer in the output layer constitutes the input of the output layer, and the output of the Conv layer in the output layer constitutes the output of the output layer.

[0083] The output of the front-end CBS layer connected to the first output of the backbone network is connected to the intermediate input of the third first combination layer in sequence. The output of the front-end CBS layer connected to the second output of the backbone network is connected to the intermediate input of the second first combination layer in sequence. The output of the front-end CBS layer connected to the third output of the backbone network is connected to the intermediate input of the first first combination layer in sequence. The output of the front-end CBS layer connected to the fourth output of the backbone network is connected to the input of the first first combination layer in sequence. The output of the third first combination layer in sequence is connected to the input of the first second combination layer in sequence. The output of the second first combination layer in sequence is connected to the intermediate input of the first second combination layer in sequence. The output of the first first combination layer in sequence is connected to the intermediate input of the second second combination layer in sequence. The output of the front-end CBS layer connected to the fourth output of the backbone network is also connected to the intermediate input of the third second combination layer in sequence. The outputs of the third first combination layer, the first second combination layer, the second second combination layer, and the third second combination layer in sequence are connected one-to-one to the inputs of each output layer. The outputs of each output layer constitute the outputs of the improved Yolov7 network structure.

[0084] In practical applications, the improved Yolov7 network structure described above has the same structure for each LSAM layer, such as... Figure 2 As shown, each LSAM layer includes a CAM module, a SAM module, a first element-wise summation module, an activation function module, a normalization module, a matrix multiplication module, and a second element-wise summation module. In each LSAM layer structure, the input terminals of the CAM module and the SAM module constitute the two input terminals of the LSAM layer, which are used to receive feature maps. The CAM module is used to calculate the channel attention weight matrix E of the received feature map. c The output, the SAM module, is used to calculate the spatial attention weight matrix E of the received feature map. s The outputs of the CAM module and the SAM module are connected to the input of the first element-wise summation module. The output of the first element-wise summation module is then connected to the input of the normalization module via the activation function module, executing E. cs =sigmoid(E C +E S The output of the normalization module is based on the spatial and channel attention weight matrix E. cs The output of the activation module is connected to the input of the matrix multiplication module. The matrix multiplication module performs the multiplication on the received E. cs The first module performs matrix multiplication on the feature map to obtain the processing result. The output of the matrix multiplication module is connected to the input of the second element-wise summation module. The second element-wise summation module performs element-wise summation on the feature map based on the processing result of the matrix multiplication module, thereby updating the feature map output.

[0085] Furthermore, in the improved Yolov7 network architecture, such as Figure 3As shown, the specific design of other object layers is as follows: Each CBS layer has the same structure, and each CBS layer includes a Conv layer, a BN layer, and a SiLU layer connected in series from its input to its output. Each MPconv layer has the same structure, where the input of the MPconv layer connects to two parallel structures. One structure connects the MaxPool layer and the CBS layer in series, and the other structure connects two CBS layers in series. The outputs of these two structures connect to the input of the Concat layer, and the output of the Concat layer constitutes the output of the MPconv layer. Each ELAN layer has the same structure. The input of the ELAN layer connects to a two-way parallel structure. The first structure consists of five CBS layers connected in series, while the second structure is a single CBS layer. The outputs of the first, third, and fifth CBS layers in the first structure, and the output of the CBS layer in the second structure, are connected to the input of the Concat layer. The output of the Concat layer is connected to the input of another CBS layer, which together form the output of the ELAN layer. Each ELAN-W layer has the same structure. In each ELAN-W layer structure, the input of the ELAN-W layer connects to a two-way parallel structure. The first structure consists of five CBS layers connected in series, while the second structure is a single CBS layer. The structure consists of a single CBS layer. In the first structure, the outputs of each CBS layer, and in the second structure, the outputs of the CBS layers, are connected to the inputs of the Concat layer. The output of the Concat layer is connected to the input of another CBS layer, whose outputs constitute the output of the ELAN-W layer. Each REPconv layer has the same structure. In each REPconv layer structure, the input of the REPconv layer is connected to three identical parallel structures. Each structure includes a series-connected Conv layer and a BN layer. The output of each structure is connected to the input of the Add layer. The output of the Add layer is connected to the input of the SiLU layer. The output of the SiLU layer... The output terminals constitute the output terminals of the REPconv layer; in the SPP layer structure, the input terminals of the SPP layer are connected to the input terminals of the first CBS layer, and the output terminals of the first CBS layer are connected to the input terminals of the second CBS layer, the input terminals of the Concat layer, and the input terminals of the three parallel MaxPool layers respectively. The input terminal of the Concat layer is also connected to the output terminals of the three parallel MaxPool layers, and the output terminal of the Concat layer is connected to the input terminal of the third CBS layer. The output terminals of the second CBS layer and the third CBS layer are connected to the input terminals of another Concat layer, and the output terminal of the other Concat layer constitutes the output terminal of the SPP layer.

[0086] In practical applications, regarding the above design scheme, TensorRT is used to accelerate the inference of the trained human detection model and the safety wearable detection model. Specifically, the trained model is first converted into an ONNX model, then loaded into TensorRT and converted into a TRRT object. Inference is then performed using the object in TRRT. Using TensorRT to perform horizontal or vertical merging between network layers significantly reduces the number of network layers, thereby accelerating the inference of deep neural networks, reducing memory resource consumption, and greatly improving inference speed.

[0087] In practical applications, the above design is further integrated with the Yolov7, Deepsort, and TensorRT code to write a detection process. In actual application processing, an alarm is generated when it is detected that a worker is not wearing safety equipment correctly.

[0088] The aforementioned technical solution, based on Yolov7 and Deepsort, designs a safety wear detection method that introduces the LSAM attention mechanism and innovatively designs an improved Yolov7 network structure. This is used to train a person detection model and a safety wear detection model. The person detection model is applied to analyze captured images to obtain person detection results. Then, the Deepsort network is applied to achieve continuous person tracking between captured frames. Finally, the safety wear detection model is applied to perform further safety wear detection for continuous person tracking. The improved network structure enhances the network's ability to perceive small targets, improving detection accuracy. Furthermore, by adding the velocity direction factor to the cost matrix during person tracking using the Deepsort network, the accuracy and robustness of target tracking are improved. This allows for better capture of key features in the image, more effective target tracking, and significantly reduced false positives and false negatives in the safety wear detection process.

[0089] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A secure wearable detection method based on Yolov7 and Deepsort, characterized in that: For the images captured in real time by the image capture device, the following steps A to F are executed in real time to achieve real-time detection of the person being photographed wearing a safety device. Step A. Based on the pre-trained person detection model with an improved Yolov7 network structure that takes the image as input and outputs the person detection results in the image, analyze the currently captured image to obtain the person detection results in the currently captured image. The person detection results include whether there are people in the image and the location of the people in the image. Then proceed to Step B. Step B. Determine whether a person exists in the current image. If so, obtain the local image of the current person in each frame of the current image and proceed to step C; otherwise, the processing of the current image ends. Step C. Determine whether there is a previous frame image that is adjacent to the current captured image and contains a partial image of the person in the historical time direction. If yes, proceed to step D; otherwise, for each frame of the current partial image of the person in the current captured image, mark each frame with a tracking mark that is different from the already marked tracking mark and is different from each other. This constitutes a new tracking mark corresponding to each frame of the current partial image of the person in the current captured image, and proceed to step F. Step D. Based on the tracking markers corresponding to the local images of each person in the previous frame, and considering the historical time direction, according to the consecutive local images of each person corresponding to each tracking marker, apply a preset feature extraction network and a Deepsort network to track and detect the local images of the current person in each frame of the current image. Specifically, for each frame of the current person's local image that has been successfully tracked, the corresponding current person's local image is marked with each tracking mark that has been successfully tracked, forming the old tracking mark corresponding to each frame of the current person's local image, and then proceeding to step E; For each frame of the current local image of the person that was not successfully tracked, a new tracking mark is marked that is different from the existing tracking mark and is different from each other, thus forming a new tracking mark corresponding to each frame of the current local image of the person, and then proceed to step F; Step E. Based on the current person's local images in each frame corresponding to the old tracking markers, for each tracking marker, based on the consecutive person's local images corresponding to the tracking marker, if the person corresponding to the tracking marker is in the correct safety wearing state, then no processing is performed on the current person's local image corresponding to the tracking marker; If the person corresponding to the tracking tag is wearing the wrong safety gear, an alarm will be triggered on the local image of the person corresponding to the tracking tag. If the person corresponding to the tracking tag is in a temporary analysis wearing state, then the following processing shall be performed: First, a pre-trained safety wear detection model with an improved Yolov7 network structure is applied. This model takes a local image of a person as input and outputs the result of whether the person in the local image is wearing various preset types of safety clothing. The model is then analyzed for the current local image of the person corresponding to the tracking marker to obtain the result of whether the person in the current local image is wearing various preset types of safety clothing. Then, it is determined whether all types of safety clothing are worn under the cumulative analysis of the safety wear detection model for the consecutive local images of the person corresponding to the tracking marker. If so, the person corresponding to the tracking marker is defined as being in a correct safety wear state, and no processing is performed on the local image of the current person corresponding to the tracking marker. Otherwise, it is further determined whether the number of frames of the consecutive local images of the person corresponding to the tracking marker has reached the preset safety detection frame limit. If so, the person corresponding to the tracking marker is defined as being in an incorrect safety wear state, and an alarm is triggered on the local image of the current person corresponding to the tracking marker. Otherwise, the person corresponding to the tracking marker is kept in a temporary analysis and wear state, and no processing is performed on the local image of the current person corresponding to the tracking marker. Step F. For each frame of the current person's local image corresponding to the new tracking marker, first apply a pre-trained safety wear detection model with an improved Yolov7 network structure. This model takes the local image as input and outputs whether the person is wearing any of the preset safety clothing items. Analyze the current person's local image to obtain the results of whether the person is wearing any of the preset safety clothing items. Then determine whether the person is wearing all of the preset safety clothing items. If yes, define the person corresponding to the tracking marker as being in a correct safety wear state, and do not process the current person's local image corresponding to the tracking marker. Otherwise, define the person corresponding to the tracking marker as being in a temporary safety wear state, and do not process the current person's local image corresponding to the tracking marker. The improved YOLOv7 network structure includes a backbone network and a head network. The backbone network consists of CBS layers, CBS layers, CBS layers, CBS layers, ELAN layers, LSAM layers, MPconv layers, ELAN layers, LSAM layers, MPconv layers, ELAN layers, LSAM layers, MPconv layers, ELAN layers, LSAM layers, MPconv layers, ELAN layers, LSAM layers, and SPP layers, which are connected in series in the data flow direction. The input of the first CBS layer constitutes the input of the improved YOLOv7 network structure. The output of the first LSAM layer constitutes the first output of the backbone network. The output of the second LSAM layer constitutes the second output of the backbone network. The output of the third LSAM layer constitutes the third output of the backbone network. The output of the SPP layer constitutes the fourth output of the backbone network. Each LSAM layer has the same structure, including a CAM module, a SAM module, a first element-wise summation module, an activation function module, a normalization module, a matrix multiplication module, and a second element-wise summation module. In each LSAM layer structure, the inputs of the CAM module and the SAM module constitute the two inputs of the LSAM layer, used to receive feature maps. The CAM module is used to calculate the channel attention weight matrix E of the received feature map. c The output, the SAM module, is used to calculate the spatial attention weight matrix E of the received feature map. s The outputs of the CAM module and the SAM module are connected to the input of the first element-wise summation module. The output of the first element-wise summation module is then connected to the input of the normalization module via the activation function module. The output of the normalization module outputs the spatial and channel attention weight matrix E. cs The output of the activation module is connected to the input of the matrix multiplication module. The matrix multiplication module performs the multiplication on the received E. cs The first module performs matrix multiplication on the feature map to obtain the processing result. The output of the matrix multiplication module is connected to the input of the second element-wise summation module. The second element-wise summation module performs element-wise summation on the feature map based on the processing result of the matrix multiplication module, thereby updating the feature map output. The detection head network head includes four front-end CBS layers, three first combination layers, three second combination layers, and four output layers. Each output of the backbone network is connected to one of the four front-end CBS layers in the detection head network head. The structures of each first combination layer are identical. Each first combination layer, from its input to its output, includes a CBS layer, an Upsample layer, a Concat layer, and an ELAN-W layer connected in series. The input of the CBS layer in each first combination constitutes the input of that first combination, and the output of the ELAN-W layer constitutes the output of that first combination. The first combination layers are connected in series according to the data flow direction. The input of the Concat layer in each first combination constitutes the intermediate input of that first combination. Each second combination layer... The structures are identical to each other. Each second combined layer includes, from its input to its output, a series of MPconv, Concat, and ELAN-W layers connected in series. The input of the MPconv layer in each second combined layer constitutes the input of the second combined layer, and the output of the ELAN-W layer in each second combined layer constitutes the output of the second combined layer. The second combined layers are connected in series in the direction of data flow. The input of the Concat layer in each second combined layer constitutes the intermediate input of each second combined layer. The structures of the output layers are identical to each other. Each output layer includes, from its input to its output, a series of REPconv and Conv layers connected in series. The input of the REPconv layer in each output layer constitutes the input of the output layer, and the output of the Conv layer in each output layer constitutes the output of the output layer. The output of the front-end CBS layer connected to the first output of the backbone network is connected to the intermediate input of the third first combination layer in sequence. The output of the front-end CBS layer connected to the second output of the backbone network is connected to the intermediate input of the second first combination layer in sequence. The output of the front-end CBS layer connected to the third output of the backbone network is connected to the intermediate input of the first first combination layer in sequence. The output of the front-end CBS layer connected to the fourth output of the backbone network is connected to the input of the first first combination layer in sequence. The output of the third first combination layer in sequence is connected to the input of the first second combination layer in sequence. The output of the second first combination layer in sequence is connected to the intermediate input of the first second combination layer in sequence. The output of the first first combination layer in sequence is connected to the intermediate input of the second second combination layer in sequence. The output of the front-end CBS layer connected to the fourth output of the backbone network is also connected to the intermediate input of the third second combination layer in sequence. The outputs of the third first combination layer, the first second combination layer, the second second combination layer, and the third second combination layer in sequence are connected one-to-one to the inputs of each output layer. The outputs of each output layer constitute the outputs of the improved Yolov7 network structure.

2. The secure wearable detection method based on Yolov7 and Deepsort according to claim 1, characterized in that: Each of the CBS layers has the same structure, and each CBS layer includes a Conv layer, a BN layer, and a SiLU layer connected in series from its input end to its output end. Each MPconv layer has the same structure. In each MPconv layer structure, the input end of the MPconv layer is connected to two parallel structures. One structure is connected in series with the MaxPool layer and the CBS layer, and the other structure is connected in series with two CBS layers. The output end of the two structures is connected to the input end of the Concat layer, and the output end of the Concat layer constitutes the output end of the MPconv layer. Each ELAN layer has the same structure. In each ELAN layer structure, the input terminal of the ELAN layer is connected to two parallel structures. The first structure is five CBS layers connected in series. The second structure is a single CBS layer. The output terminals of the first, third, and fifth CBS layers in the first structure and the CBS layer output terminal in the second structure are connected to the input terminal of the Concat layer. The output terminal of the Concat layer is connected to the input terminal of another CBS layer. The output terminal of this CBS layer constitutes the output terminal of the ELAN layer. Each ELAN-W layer has the same structure. In each ELAN-W layer structure, the input of the ELAN-W layer is connected to two parallel structures. The first structure is five CBS layers connected in series in sequence. The second structure is a single CBS layer. The outputs of each CBS layer in the first structure and the outputs of the CBS layer in the second structure are connected to the input of the Concat layer. The output of the Concat layer is connected to the input of another CBS layer. The output of this CBS layer constitutes the output of the ELAN-W layer. Each REPconv layer has the same structure. In each REPconv layer structure, the input end of the REPconv layer is connected to three identical parallel structures. Each structure includes a series Conv layer and a BN layer. The output end of each structure is connected to the input end of the Add layer. The output end of the Add layer is connected to the input end of the SiLU layer. The output end of the SiLU layer constitutes the output end of the REPconv layer. In the SPP layer structure, the input of the SPP layer is connected to the input of the first CBS layer. The output of the first CBS layer is connected to the input of the second CBS layer, the input of the Concat layer, and the input of three parallel MaxPool layers. The input of the Concat layer is also connected to the output of the three parallel MaxPool layers. The output of the Concat layer is connected to the input of the third CBS layer. The outputs of the second and third CBS layers are connected to the input of another Concat layer. The output of the other Concat layer constitutes the output of the SPP layer.

3. The secure wearable detection method based on YOLOv7 and Deepsort according to claim 1 or 2, characterized in that: Regarding the person detection model, based on a preset number of sample images of each frame with known positions of person detection boxes in the image, the sample images are used as input and the positions of person detection boxes in the sample images are used as output. The model is trained on an improved Yolov7 network structure to obtain the person detection model. Regarding the safety wear detection model, it is based on a preset number of sample images of people in each frame, each containing a known number of frames showing whether a person is wearing a preset type of safety clothing. The model is trained on an improved Yolov7 network structure, with the sample images as input and the output showing whether a person is wearing a preset type of safety clothing.

4. The secure wearable detection method based on YOLOv7 and Deepsort according to claim 3, characterized in that: For each frame of sample image, rotation, cropping, and adjustment of brightness, contrast, saturation, and hue are performed separately to obtain new sample images for each operation, thereby expanding each frame of sample images. Then, based on the expanded frame of sample images, the improved Yolov7 network structure is trained to obtain a person detection model. For each frame of the human sample image, new human sample images are obtained by rotating, cropping, and adjusting the brightness, contrast, saturation, and hue of the image. This expands each frame of the human sample image. Based on the expanded human sample images, the improved Yolov7 network structure is trained to obtain a safety wear detection model.

5. The secure wearable detection method based on Yolov7 and Deepsort according to claim 1, characterized in that: In step D, tracking markers are used for each local image of a person in the previous frame. and partial images of the current person in each frame of the currently captured image. Follow steps D1 to D4 below to complete the tracking of the markers in the previous frame. The corresponding partial image of the person and the current partial image of the person The detection of whether the tracking is between frames is used to achieve tracking and detection of local images of the current person in each frame of the currently captured image; Step D1. First, apply the Kalman filter in the Deepsort network to obtain the tracking markers in the previous frame image. The corresponding partial image of the person is predicted within the current captured image. Then, a pre-trained feature extraction model for extracting preset features from the image is applied to extract the preset feature vectors of the predicted partial image of the person. Simultaneously extract a local image of the current person. The preset feature vectors of the detection box Finally, use the following formula: ； Get tracking tags The corresponding local image prediction bounding box of the person and the current local image of the person Motion features of Mahalanobis distance between detection boxes And proceed to step D2; where, Indicates tracking mark The corresponding local image prediction bounding box of the person and the current local image of the person The covariance matrix between the detection boxes; Step D2. Based on tracking tags The appearance description features corresponding to each consecutive local image of a person and a partial image of the current person Appearance description features of the detection box According to the following formula: ； Get tracking tags The corresponding consecutive partial images of each person and the current partial image of the person The minimum cosine value between the appearance description features Then proceed to step D3; where, Indicates tracking mark The corresponding consecutive partial images of each person The appearance description features corresponding to a partial image of a person. express transpose; Step D3. Based on the preset weights According to the following formula: ； get and The cost of association between them Then proceed to step D4; Step D4. According to the Hungarian algorithm, use the following formula: ； Obtain the tracking markers in the previous frame image The corresponding partial image of the person and the current partial image of the person The cost of association between them In conjunction with a preset correlation cost threshold, the tracking markers in the previous frame image are determined. The corresponding partial image of the person and the current partial image of the person Whether there is a correlation between them, in which, Indicates the preset weighting factor. Indicates the tracking marker in the previous frame. The trajectory direction of the corresponding partial image of the person. Indicates the tracking marker in the previous frame. The corresponding partial image of the person and the current partial image of the person The direction of the trajectory formed by the connection and The directional difference between them is expressed as the absolute difference in radians.

Images