Wearing detection method and device, equipment and storage medium

Abstract

The invention provides a wearing detection method and device, equipment and a storage medium, and the method comprises the steps: extracting a feature set of each person in a to-be-detected image through a multi-task neural network model, the feature set comprises a person frame position, person identification information, a person head frame position, a safety helmet frame position, a tool frame position, a non-tool frame position and a person upper body frame position; determining a target identifier corresponding to each person based on the person identifier information and the person frame position; determining whether each person wears a safety helmet or not based on the head frame position and the safety helmet frame position; and determining whether each person wears the tool or not based on the tool frame position and the upper body frame position of the person. According to the embodiment of the invention, the feature set of each person in the to-be-detected image is simultaneously detected and recognized through the multi-task neural network model, detection of the illegal wearing behavior is realized based on the feature sets, and the detection efficiency is improved.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a wearable detection method, device, equipment, and storage medium. Background Technology

[0002] Detecting improper use of safety equipment is a crucial aspect of safety inspections. Especially at construction sites and factory production areas, in the event of incidents such as falling objects or machinery malfunctions, workers not wearing the required safety helmets, protective gear, and other appropriate protective equipment pose a significant safety hazard. Therefore, timely detection of improper equipment use by on-site workers can effectively prevent accidents and improve the efficiency of safe production operations.

[0003] Related technologies typically employ manual visual inspection, sensor detection, and visual inspection to determine whether workers are wearing safety equipment improperly. Manual visual inspection primarily involves safety supervisors reviewing monitoring system footage or conducting on-site inspections to detect improper use. Sensor-based detection methods mainly determine the wearing status of safety equipment by analyzing the status of sensors installed on the safety equipment.

[0004] Manual visual inspection is inefficient and costly due to the limitations of human vision, and is prone to missed detections; sensor-based detection is also expensive. Summary of the Invention

[0005] This application proposes a method, device, equipment, and storage medium for detecting wearable devices, which can solve the technical problems of low efficiency and high labor costs in the current detection of unauthorized wearable behaviors of on-site workers.

[0006] The first aspect of this application provides a wearable detection method, including:

[0007] A multi-task neural network model is used to extract the feature set of each person in the image to be detected. The feature set includes the person's border position, person identification information, head border position, safety helmet border position, work clothes border position, non-work clothes border position, and the upper half of the person's border position.

[0008] Based on the personnel identification information and the personnel border position, determine the target identifier corresponding to each personnel;

[0009] Based on the first relative positional relationship between the head frame position and the helmet frame position, it is determined whether each person is wearing a helmet;

[0010] determine whether each of the persons wears a safety helmet based on a first relative position relationship between the head bounding box position and the safety helmet bounding box position.

[0011] Embodiments of the second aspect of the present application provide a wearing detection apparatus, comprising:

[0012] The extraction module is configured to extract a feature set of each of the persons in the to-be-detected image by using the multi-task neural network model, the feature set comprising a person bounding box position, person identification information, a head bounding box position, a safety helmet bounding box position, a tool bounding box position, a non-tool bounding box position, and a person upper body bounding box position.

[0013] The determination module is configured to determine a target identification corresponding to each of the persons based on the person identification information and the person bounding box position.

[0014] The determination module is configured to determine whether each of the persons wears a safety helmet based on a first relative position relationship between the head bounding box position and the safety helmet bounding box position.

[0015] The determination module is configured to determine whether each of the persons wears a tool based on a second relative position relationship between the tool bounding box position and the person upper body bounding box position.

[0016] Embodiments of the third aspect of the present application provide a computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the method of the first aspect.

[0017] Embodiments of the fourth aspect of the present application provide a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method of the first aspect.

[0018] The technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0019] The application provides a wearing detection method, device and storage medium. The method comprises the following steps: extracting a feature set of each person in a to-be-detected image by using a multi-task neural network model, wherein the feature set comprises a person bounding box position, person identification information, a head bounding box position, a safety helmet bounding box position, a tool bounding box position, a non-tool bounding box position and a person upper body bounding box position; determining a target identification corresponding to each person based on the person identification information and the person bounding box position; determining whether each person wears a safety helmet based on the head bounding box position and the safety helmet bounding box position; and determining whether each person wears a tool based on the tool bounding box position and the person upper body bounding box position. The embodiment of the application simultaneously detects and identifies the feature set of each person in the to-be-detected image by using the multi-task neural network model, and realizes the detection of the illegal wearing behavior based on the feature set, thereby improving the detection efficiency.

[0020] Additional aspects and advantages will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and / or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0021] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The detailed description is made with reference to the accompanying drawings.

[0022] In the drawings:

[0023] Figure 1 A wearing detection method flow chart provided by an embodiment of the application is shown;

[0024] Figure 2 A structure schematic diagram of a multi-task neural network model provided by an embodiment of the application is shown;

[0025] Figure 3 A human key point schematic diagram provided by an embodiment of the application is shown;

[0026] Figure 4 A structure schematic diagram of a wearing detection device provided by an embodiment of the application is shown;

[0027] Figure 5 A structure schematic diagram of a computing device provided by an embodiment of the application is shown;

[0028] Figure 6 A schematic diagram of a storage medium provided by an embodiment of the application is shown. DETAILED DESCRIPTION

[0029] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it is understood that the present application can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0030] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present application should be understood as the general meaning understood by those skilled in the art to which the present application belongs.

[0031] In addition to the foregoing background, the presence or absence of a violation of the wearing behavior of the worker can also be determined by visual detection. The visual detection-based method mainly uses traditional machine learning methods or deep learning technology to realize the detection of the violation of the wearing behavior of the worker. This kind of method usually detects the worker and the safety helmet in the image first, and then determines whether the worker wears the safety helmet according to the position relationship between the safety helmet and the worker's frame. In addition, the color, depth and other features of the worker subgraph are identified and analyzed to determine whether the worker wears the work clothes. However, when the human body is in a squatting or prone position, etc., the method of simply judging according to the position relationship of the frame may cause false detection and missed detection. Therefore, some optimization schemes add human body posture estimation on the basis of worker detection to determine the position of the human head and torso, and then further identify and analyze the wearing behavior of the worker. At the same time, in this cascading detection and identification process, the detection effect of the later stage needs to rely on the detection result of the former stage, so this method has the problems of low efficiency and instability.

[0032] To solve the above technical problems, the present application provides a wearing detection method, device, equipment and storage medium, comprising: extracting a feature set of each worker in a to-be-detected image by using a multi-task neural network model, the feature set comprising a worker frame position, worker identification information, a human head frame position, a safety helmet frame position, a work clothes frame position, a non-work clothes frame position and a worker upper body frame position; determining a target identification corresponding to each worker based on the worker identification information and the worker frame position; determining whether each worker wears a safety helmet based on the human head frame position and the safety helmet frame position; and determining whether each worker wears work clothes based on the work clothes frame position and the worker upper body frame position. The present application embodiment simultaneously detects and identifies the feature set of each worker in the to-be-detected image by using a multi-task neural network model, and realizes the detection of the violation of the wearing behavior based on the feature set, thereby improving the detection efficiency.

[0033] The wearing detection method of the present application can be executed by a computing device, which can be a server, for example, one server, multiple servers, a server cluster, a cloud computing platform, etc. Optionally, the computing device can also be a terminal device, for example, a mobile phone, a tablet computer, a game console, a portable computer, a desktop computer, a billboard, an all-in-one machine, etc. The present application does not limit the device type and the number of devices of the computing device.

[0034] A wearing detection method according to an embodiment of the present application is described below in combination with the accompanying drawings.

[0035] Referring to Figure 1 The method specifically includes the following steps:

[0036] S101, extracting a feature set of each person in the to-be-detected image by using a multi-task neural network model.

[0037] The feature set includes a person bounding box position, person identification information, a head bounding box position, a safety helmet bounding box position, a tool bounding box position, a non-tool bounding box position, and a person upper body bounding box position.

[0038] The to-be-detected image can be a video frame of a monitoring video of a place.

[0039] In some embodiments, the to-be-detected image at the current time t can be obtained by extracting frames from a monitoring video stream at an interval.

[0040] It can be understood that the present application mainly detects whether a person wears in violation of regulations, and therefore, the feature set corresponding to each person can be extracted from the to-be-detected image.

[0041] In some embodiments, detecting whether a person wears in violation of regulations mainly detects whether the person wears a safety helmet and wears safety equipment such as tool, and generally, the tool on the upper body of the person is mainly detected.

[0042] In order to detect whether the person wears a safety helmet and wears safety equipment such as tool, the feature information such as the person bounding box position, the person identification information, the head bounding box position, the safety helmet bounding box position, the tool bounding box position, the non-tool bounding box position, and the person upper body bounding box position needs to be obtained, so as to detect whether the person wears a safety helmet and wears safety equipment such as tool based on the feature information.

[0043] S102, determining a target identification corresponding to each person based on the person identification information and the person bounding box position.

[0044] In some embodiments, to determine whether each person wears a safety helmet and whether each person wears safety equipment such as work clothes, it is necessary to identify each person first, and therefore, the target identification corresponding to each person can be determined based on the personnel identification information and the position of the personnel bounding box.

[0045] In some embodiments, after the position of the personnel bounding box and the personnel identification information are identified, the position of the personnel bounding box and the personnel identification information can be matched based on a preset correspondence, so that the target identification corresponding to each person can be determined.

[0046] In some embodiments, a multi-target tracking algorithm can also be used to perform feature matching between the current frame and the previous frame to determine which targets are the same target, and for the successfully matched targets, the trajectory information, including the position, speed, and appearance features, is updated. For the unmatched targets, a new trajectory can be created. The ID number and the corresponding bounding box position of each target are outputted for subsequent analysis or application.

[0047] S103, determining whether each person wears a safety helmet based on the first relative position relationship between the position of the head bounding box and the position of the safety helmet bounding box.

[0048] S104, determining whether each person wears work clothes based on the second relative position relationship between the position of the work clothes bounding box and the position of the upper body bounding box of the person.

[0049] The first relative position relationship and the second relative position relationship can be the area of mutual coverage or the intersection over union value.

[0050] In some embodiments, determining whether each person wears a safety helmet based on the first relative position relationship between the position of the head bounding box and the position of the safety helmet bounding box includes: sequentially calculating the first intersection over union value of the position of the head bounding box and the position of the safety helmet bounding box of each person; and if the first intersection over union value is greater than a first preset intersection over union value, it is determined that the corresponding person wears a safety helmet.

[0051] The first intersection over union value of the position of the head bounding box and the position of the safety helmet bounding box corresponding to a certain person is calculated if the category corresponding to the person is detected to include a safety helmet.

[0052] The intersection over union value can be calculated by the following formula:

[0053]

[0054] The IOU is the first intersection over union value, S1 is the position of the head bounding box, S2 is the position of the safety helmet bounding box, Area(S1∩S2) is the area of the intersection of the position of the head bounding box and the position of the safety helmet bounding box, and Area(S1∪S2) is the area of the union of the position of the head bounding box and the position of the safety helmet bounding box.

[0055] Furthermore, if the first crossover ratio is greater than the first preset crossover ratio, then it is determined that the corresponding person is wearing a safety helmet. The first preset crossover ratio can be flexibly set based on the actual situation, and will not be elaborated here.

[0056] In some embodiments, if the first crossover-union ratio is not greater than the first preset crossover-union ratio, it is determined that the corresponding person is not wearing a safety helmet or is not wearing the safety helmet properly.

[0057] In some embodiments, determining whether each person is wearing work clothes based on the second relative positional relationship between the work clothes border position and the upper half of the person's body frame position includes: if the clothing category of the first person is determined to be work clothes, then calculating the second intersection-union ratio (IUU) between the work clothes border position and the upper half of the person's body frame position, where the first person is any one of the multiple people in the image to be detected; if the second IUU is greater than a second preset IUU, then determining that the corresponding person is wearing work clothes; if the clothing category of the first person is determined to be work clothes and non-work clothes, then calculating the third IUU between the work clothes border position and the upper half of the person's body frame position; if the second IUU is greater than a third preset IUU, then determining that the corresponding person is wearing work clothes.

[0058] The calculation methods for the second and third crossover ratios are similar to those for the second crossover ratio, and will not be repeated here.

[0059] It is understandable that the clothing category for a certain person can be workwear, non-workwear, or both.

[0060] If the clothing category of the first person is determined to be work clothes, then the second intersection-union ratio of the work clothes border position and the upper half of the person's side frame position is calculated. If the second intersection-union ratio is greater than the second preset intersection-union ratio, then the corresponding person is determined to be wearing work clothes. If the clothing category of the first person is determined to be non-work clothes, then the corresponding person is determined not to be wearing work clothes. If the clothing category of the first person is determined to be both work clothes and non-work clothes, then the third intersection-union ratio of the work clothes border position and the upper half of the person's side frame position is calculated. If the third intersection-union ratio is greater than the third preset intersection-union ratio, then the corresponding person is determined to be wearing work clothes.

[0061] In some embodiments, if it is determined that the first person's clothing category is work clothes and the second crossover ratio is not greater than the second preset crossover ratio, then it is determined that the person's work clothes are not worn correctly.

[0062] If it is determined that the first person's clothing category is work clothes and non-work clothes, and the third cross union ratio is greater than the third preset cross union ratio, then it is determined that the person's work clothes are not worn properly, or the person is not wearing work clothes.

[0063] The first preset intersection-union ratio, the second preset intersection-union ratio, and the third preset intersection-union ratio can be the same or different, and can be flexibly set based on the actual situation.

[0064] In some embodiments, the final determination of whether a person has violated the rules of wearing protective clothing is based on the detection results of multiple frames. For example, if a person is detected not wearing a safety helmet or work clothes at multiple consecutive moments, it is considered that the person has violated the rules of wearing protective clothing; otherwise, it is not.

[0065] The number of multiple moments can be flexibly set based on the actual situation, such as 3, 4, 5, etc.

[0066] This application proposes a method, apparatus, device, and storage medium for detecting clothing use, comprising: extracting a feature set for each person in an image to be detected using a multi-task neural network model; the feature set includes the person's border position, person identification information, head border position, safety helmet border position, workwear border position, non-workwear border position, and upper half of the person's body border position; determining the target identifier corresponding to each person based on the person identification information and the person's border position; determining whether each person is wearing a safety helmet based on the head border position and the safety helmet border position; and determining whether each person is wearing workwear based on the workwear border position and the upper half of the person's body border position. This application's embodiment simultaneously detects and identifies the feature set of each person in the image to be detected using a multi-task neural network model, and detects unauthorized clothing use based on the feature set, thus improving detection efficiency.

[0067] In some embodiments, the multi-task neural network model includes a pyramid network and a head network. The multi-task neural network model is used to determine the feature set of each person in the image to be detected, including: inputting the image to be detected into the pyramid network and outputting a target feature map after fusing feature images of different scales corresponding to the image to be detected; inputting the target feature map into the head network to obtain the feature set of each person in the image to be detected.

[0068] In some embodiments, in order to fuse shallow and deep features of the input image, the pyramid network may include a backbone network and a neck network.

[0069] In some embodiments, the model structure of the multi-task neural network model is as follows: Figure 2 As shown, the multi-task neural network model consists of three parts: the backbone network, the neck network, and the head network.

[0070] The backbone network is used to extract depth features from the input image. Image I to be detected. input ∈R H×W×3Here, H and W represent the length and width of the input image, respectively, and the number "3" represents the RGB three channels of the input image. First, the image passes through a convolutional layer (Conv) with a kernel of 7×7, 32 output channels, and a stride of 2 to calculate the depth feature map. Then, through four convolutional module layers (ConvBlock), depth feature maps at four different scales of the image are obtained. ConvBlock layer structure as follows Figure 1 As shown in the bottom right corner, a residual network design is used. Each ConvBlock layer contains one average pooling layer (AVGPool) with a pooling kernel of 3×3 and a stride of 2, and four convolutional kernels of 3×3, with an output channel of C. out The Conv layers have four sublayers. The first Conv layer has a step size of 2, while the other three Conv layers have a step size of 1. The output channels C of the four ConvBlock layers are... out The values ​​are 64, 128, 256, and 512 respectively.

[0071] The neck network is used to fuse shallow and deep features of the input image, enhancing the effective information of the image's depth features. The neck network employs a Feature Pyramid Network (FPN) structure, and the specific steps are as follows:

[0072] The input feature map C5 is processed through a deconvolutional layer (DeConv) with a kernel size of 3×3, an output channel size of 256, and a stride size of 2 to obtain the final feature map.

[0073] Feature maps C4 and P5 are summed at their points and then passed through a DeConv layer with a 3×3 kernel, 128 output channels, and a stride of 2 to obtain the feature map.

[0074] Feature maps C3 and P4 are summed at their points and then passed through a DeConv layer with a 3×3 kernel, 64 output channels, and a stride of 2 to obtain the final feature map.

[0075] The feature maps C2 and P3 are summed at their points and then passed through a Conv layer with a 3×3 kernel, 128 output channels, and a stride of 1 to obtain the target feature map.

[0076] Furthermore, the target feature map is input into the head network to obtain the feature set of each person in the image to be detected.

[0077] In some embodiments, the target feature map is input into the head network to obtain the feature set of each person in the image to be detected, including: determining the category corresponding to each person in the target feature map, the confidence level of each category, and the center point position of each person through the target classification network of the head network, wherein the category corresponding to each person includes any one of: person, head, safety helmet, work clothes, and non-work clothes; determining the target category corresponding to each person from the categories corresponding to each person based on the confidence level of each category; determining the size corresponding to each target category corresponding to each person through the target bounding box network of the head network; determining the position of at least one key point corresponding to each person through the target key point network of the head network, wherein at least one key point includes at least one of head, left shoulder, right shoulder, left hip, and right hip; determining the position of the upper half of the person's body frame corresponding to each person based on the position of at least one key point corresponding to each person; determining the person identification information corresponding to each person through the target identifier extraction network of the head network; and determining the feature set of each person in the image to be detected based on the branch networks in the head network.

[0078] in, Figure 3 The head network shown is used to implement multi-task learning and computation, and mainly consists of four branches:

[0079] The first branch is the target classification task branch, used to calculate the category, confidence score, and center point location of the target to be detected. Here, the target categories include five types: personnel, heads, safety helmets, work clothes, and non-work clothes. The input target feature map P2 is processed by a Conv layer with a 3×3 kernel, 5 output channels, and a stride of 1, followed by a sigmoid function layer, to output the target category feature map. F cls The five channels correspond to five categories: personnel, head, safety helmet, work clothes, and non-work clothes.

[0080] The second branch is the target bounding box regression branch, used to regress and calculate the bounding box size of the target to be detected. The input target feature map P2 is processed by a Conv layer with a 3×3 kernel, 2 output channels, and a stride of 1, followed by an Exp function layer, to output the target bounding box feature map. A box The two channels correspond to the width and height of the target to be detected, respectively.

[0081] The third branch is the target keypoint regression task branch, used to regress and calculate the keypoint locations of the target to be detected. Here, keypoint locations are only calculated for targets categorized as "persons," such as human keypoints. Figure 3As shown, this includes five key points: the center of the head, left shoulder, right shoulder, left hip, and right hip. The input target feature map P2 is processed through a Conv layer with a 3×3 kernel, 10 output channels, and a stride of 1, resulting in the output human key point feature map. F kpt The 10 channels correspond to the X-axis and Y-axis offsets of the five key points of the personnel target from the target center point.

[0082] The fourth branch is the target ID feature extraction branch, used to extract the ID feature vector of the target to be detected. Here, only the ID feature vector is calculated for targets of the category of "person". The input target feature map P2 is processed by a Conv layer with a 3×3 kernel, 5 output channels, and a stride of 1, followed by a Softmax function layer, to output the person ID feature map. The feature vector dimension for each person is 256.

[0083] like Figure 2 The convolutional architecture shown (ConvBlock), including the Conv and Deconv layers in the backbone and neck networks, requires the calculation of feature normalization (BatchNorm) and activation function processing (ReLU) after performing convolution or deconvolution on the input features.

[0084] In some embodiments, based on Figure 2 The multi-task neural network model shown extracts the feature set of each person in the image to be detected through the following steps:

[0085] Step 1: Given the image to be detected First, I t Resize the image to 800×800 pixels, then divide all pixel values ​​of the resized image by 255 to obtain the normalized image i. input ∈R 800×800×3 .

[0086] Step 2: Transfer the preprocessed image I input The input is fed into the multi-task neural network model described above, and the output after calculation is the target category feature map F. cls ∈R 200×200×5 Target bounding box feature map F box ∈R 200×200×2 Human body key point feature map F kpt ∈R 200×200×10 Personnel ID Feature Map F id ∈R 200×200×256 .

[0087] Step 3: Extract F clsThe set of elements extracted from the dataset consists of the indices and values ​​of all elements whose confidence level is greater than a preset confidence threshold. Each element corresponds to a detection target, N represents the total number of detected targets, and i represents the target index. i ,y i ) represents the coordinates of the center point of the i-th target, c i ρ represents the category of the i-th target. i =F cls [y i ,x i ,c i ] represents the confidence level of the i-th target.

[0088] Step 4: Based on the target center point coordinate set extracted in Step 3 From F box Extract the element value at the corresponding position point, and denote it as . Where w i =F box [y i ,x i [,0] and h i =F box [y i ,x i [1] represents the width and height of the border of the i-th target, respectively.

[0089] Step 5: Based on the target center point coordinate set extracted in Step 3 From F kpt Extract the element value at the corresponding position point, and denote it as . Where (dx ik ,dy ik )=(F kpt [y i ,x i ,2k],F kpt [y i ,x i [2k+1] represents the k-th key point of target i (0: center of head, 1: left shoulder, 2: right shoulder, 3: left hip, 4: right hip) and its center point (x). i ,y i The coordinate offset of the target key points. Furthermore, the coordinate set of the target key points is... Where (x) ik ,y ik )=(x i +dx ik ,y i +dy ik ) represents the coordinates of the kth key point of target i.

[0090] Step 6: Based on the target center point coordinate set extracted in Step 3 From F id Extract the element value at the corresponding position point, and denote it as . Among them FT i =F id [y i ,x i ,∶]∈R 256 This represents the ID feature vector of the i-th target.

[0091] Step 7: Through the above four steps 3, 4, 5, and 6, N targets are extracted from the feature map of the head network. These targets overlap to some extent, so the NMS method is further used to filter the target bounding boxes, leaving M targets after filtering.

[0092] Step 8: Map the target center point coordinates, bounding box width and height, and key point coordinates detected in Step 7 from the feature map back to the input image I. t The input image I is obtained. t Target set for detection Where, x′ i =x i ×r x y′ i =y i ×r y w′ i =w i ×r x , h′ i =h i ×r y ,r′ ik =r ik ×r x y′ ik =y ik ×r y , Based on the above calculations, the bounding box coordinates (x′) of each target i can be obtained. i ,y′ i ,w′ i ,h′ i Category c i Confidence level ρ i Furthermore, if the category of target i is personnel, the head region (x) of that personnel can be obtained. i_head ,y i_head ,w i_head ,h i_head ) and the upper body area (x i_body ,y i_body ,w y_body ,h i_body),in

[0093] x i_head =x′ i0 ,

[0094] y i_head =y′ i0 ,

[0095]

[0096] Finally from input image I t The target set obtained by detection is For target i that is not a personnel category, its (x i_head ,y i_head ,w i_head ,h i_head ,x i_body ,y i_body ,w i_body ,h i_body FT i The values ​​of all of them are 0.

[0097] This allows for the extraction of feature sets for each person in the image to be detected based on a multi-task neural network model.

[0098] In some embodiments, the target identifier corresponding to each person is determined based on the person identification information and the person border position, including: obtaining the person identification information and the person border position of each person in the detection image at the previous moment; calculating the similarity between the person border position of each person in the detection image at the previous moment and the person border position of each person in the image to be detected; determining the target border position based on the similarity; and outputting the target border and the target identifier corresponding to the target border.

[0099] In some embodiments, determining the target identifier corresponding to each person based on person identification information and person border position can be specifically implemented as follows:

[0100] Use object detection algorithms (such as YOLO, SSD, etc.) to detect the bounding boxes of all objects in the current frame and extract the feature vector of each object.

[0101] If there were also detected targets in the previous time step, the feature vectors of those targets are extracted in the same way.

[0102] Calculate the similarity between the target feature vectors at the current time step and the previous time step. Commonly used similarity measurement methods include:

[0103] Cosine similarity: measures the degree of similarity between two vectors in a direction.

[0104] Euclidean distance: measures the distance between two vectors in space.

[0105] Mahalanobis distance: It takes into account the covariance structure of the data distribution and is suitable for situations where the dimensional correlation of the feature space is different.

[0106] Motion models (such as Kalman filters) can be used to predict the possible location of a target detected in the previous time step in the current time step. This helps to narrow down the search range for feature matching and improve matching efficiency.

[0107] Associative algorithms (such as the Hungarian algorithm, the linear assignment problem, etc.) are used to solve feature matching problems. These algorithms attempt to find the optimal match between the target at the current time step and the target at the previous time step, in order to minimize the total matching cost (such as the negative value of distance or similarity).

[0108] For a successfully matched target, update its trajectory information, including location, speed, and appearance features.

[0109] For unmatched targets, it may be necessary to create new tracks, or if old tracks are not detected for several consecutive frames, they may need to be deleted.

[0110] For each successfully matched target, assign or update its target identifier. The target identifier can be statically assigned or dynamically generated, depending on the system design.

[0111] Output the target identifier and corresponding bounding box location for each target, so that subsequent analysis or application can be performed.

[0112] In some embodiments, the multi-task neural network model is pre-trained. The training process of the multi-task neural network model includes: acquiring a training dataset, which includes multiple sample images, category labels, identification information, key point location labels, and bounding box positions corresponding to each person in each sample image; inputting the training dataset into the multi-task neural network model to obtain the predicted category, predicted identification information, predicted key point positions, and predicted bounding box positions corresponding to each person in each sample image; determining a first loss function value based on the category labels and predicted categories; determining a second loss function value based on the identification information and predicted identification information; determining a third loss function value based on the key point location labels and predicted key point positions; determining a fourth loss function value based on the bounding box positions corresponding to each category label and the predicted bounding box positions corresponding to each predicted category; adjusting the model parameters of the multi-task neural network model based on the first, second, third, and fourth loss function values, and continuing training until the preset training completion conditions are met to obtain a trained multi-task neural network model.

[0113] The training dataset can be obtained as follows: Frames are extracted from the video stream of the monitoring camera at the work site, resulting in more than 20,000 valid images. These images are then labeled using annotation software such as Labelme and Labelimage. During annotation, the categories and bounding boxes of the targets in the images need to be labeled. Category labels are divided into personnel, heads, safety helmets, work clothes, and non-work clothes. Additionally, for targets categorized as personnel, five key points—the center point of the head, left shoulder, right shoulder, left hip, and right hip—and their ID numbers also need to be labeled.

[0114] Furthermore, the training set is input into the multi-task neural network model to obtain the predicted category, predicted identifier information, predicted key point location, and predicted bounding box location corresponding to each person in each sample image.

[0115] The loss function value of each network branch is calculated based on the predicted value and the label. After calculating each loss function value, the hyperparameter corresponding to each loss function value is determined. Based on the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value, as well as the hyperparameter corresponding to each loss function value, the total loss function value is determined.

[0116] In some embodiments, the total loss function value can be calculated using the following formula:

[0117] L total =L cls +αL box +βL kpt +γL id

[0118] Among them, L total L represents the total loss function value. cls Let L be the first loss function value. box Let L be the value of the second loss function, α be the hyperparameter corresponding to the value of the second loss function, and L be the value of the second loss function. kpt Let L be the value of the third loss function, β be the hyperparameter corresponding to the value of the third loss function, and L be the value of the third loss function. id γ is the value of the fourth loss function, and γ is the hyperparameter corresponding to the value of the fourth loss function.

[0119] The hyperparameters corresponding to the second, third, and fourth loss function values ​​can all be 0.1, or other values. Of course, the hyperparameters corresponding to the first, second, third, and fourth loss function values ​​can also be other values, which will not be elaborated upon here.

[0120] The preset training completion conditions can be that the number of training sessions reaches a training session threshold or the total loss function value is less than the preset loss function value. The training session threshold and the preset loss function value can be flexibly set based on the actual situation, and will not be elaborated here.

[0121] This application also provides a wearable detection device for performing the wearable detection method provided in any of the above embodiments. Figure 4 As shown, the device includes an extraction module 401 and a determination module 402.

[0122] The extraction module 401 is used to extract the feature set of each person in the image to be detected using a multi-task neural network model. The feature set includes the person's border position, person identification information, head border position, safety helmet border position, work clothes border position, non-work clothes border position, and the upper half of the person's border position.

[0123] The determining module 402 is used to determine the target identifier corresponding to each person based on the person identification information and the person border position;

[0124] The determining module 402 is used to determine whether each person is wearing a safety helmet based on the first relative positional relationship between the head frame position and the safety helmet frame position;

[0125] The determining module 402 is used to determine whether each person is wearing work clothes based on the second relative positional relationship between the position of the work clothes frame and the position of the upper half of the person's frame.

[0126] This application proposes a method, apparatus, device, and storage medium for detecting clothing use, comprising: extracting a feature set for each person in an image to be detected using a multi-task neural network model; the feature set includes the person's border position, person identification information, head border position, safety helmet border position, workwear border position, non-workwear border position, and upper half of the person's body border position; determining the target identifier corresponding to each person based on the person identification information and the person's border position; determining whether each person is wearing a safety helmet based on the head border position and the safety helmet border position; and determining whether each person is wearing workwear based on the workwear border position and the upper half of the person's body border position. This application's embodiment simultaneously detects and identifies the feature set of each person in the image to be detected using a multi-task neural network model, and detects unauthorized clothing use based on the feature set, thus improving detection efficiency.

[0127] In some embodiments, the extraction module 401 is specifically used for:

[0128] The image to be detected is input into the pyramid network, and the target feature map is output after fusing the feature images of different scales corresponding to the image to be detected.

[0129] The target feature map is input into the head network to obtain the feature set of each person in the image to be detected.

[0130] In some embodiments, the extraction module 401 is further specifically used for:

[0131] The target classification network of the head network determines the category corresponding to each person in the target feature map, the confidence of each category, and the center point position of each person. The category corresponding to each person includes any one of the following: person, head, safety helmet, work clothes and non-work clothes.

[0132] Based on the confidence level of each category, the target category corresponding to each person is determined from the categories corresponding to each person;

[0133] The target bounding network of the head network determines the size corresponding to the target category of each person.

[0134] The head network determines the location of at least one key point for each person through the target key point network, wherein the at least one key point includes at least one of the following: head, left shoulder, right shoulder, left hip, and right hip.

[0135] Based on the position of at least one key point of each person, determine the position of the upper half of the body frame of each person.

[0136] The target identifier extraction network of the head network is used to determine the personnel identifier information corresponding to each person;

[0137] Based on each branch network in the head network, the feature set of each person in the image to be detected is determined.

[0138] In some embodiments, the determining module 402 is specifically used for:

[0139] Obtain the person identification information and person border position for each person in the detected image at the previous moment;

[0140] Calculate the similarity between the bounding box position of each person in the detected image at the previous time step and the bounding box position of each person in the image to be detected;

[0141] The target bounding box position is determined based on the similarity.

[0142] Output the target bounding box and the target identifier corresponding to the target bounding box.

[0143] In some embodiments, the determining module 402 is further specifically used for:

[0144] Calculate the first intersection-union ratio (IUU) of the head frame position and the safety helmet frame position for each person in turn;

[0145] If the first crossover-union ratio is greater than the first preset crossover-union ratio, then it is determined that the corresponding person is wearing a safety helmet.

[0146] In some embodiments, the determining module 402 is further specifically used for:

[0147] If the clothing category of the first person is determined to be work clothes, then the second intersection-union ratio of the work clothes border position and the upper half of the person's side border position is calculated. The first person is any one of the multiple people in the image to be detected.

[0148] If the second crossover ratio is greater than the second preset crossover ratio, then it is determined that the corresponding personnel are wearing work clothes;

[0149] If it is determined that the clothing category of the first person is work clothes and non-work clothes, then calculate the third intersection-union ratio of the work clothes border position and the upper half of the person's body frame position;

[0150] If the second crossover ratio is greater than the third preset crossover ratio, then it is determined that the corresponding personnel are wearing work clothes.

[0151] In some embodiments, the multi-task neural network model is pre-trained, and the training process of the multi-task neural network model includes:

[0152] Obtain a training dataset, which includes multiple sample images, category labels for each person in each sample image, identification information, key point location labels, and bounding box positions for each category label;

[0153] The training set is input into a multi-task neural network model to obtain the predicted category, predicted identification information, predicted key point position, and predicted bounding box position for each person in each sample image.

[0154] The value of the first loss function is determined based on the category label and the predicted category;

[0155] The second loss function value is determined based on the identification information and the predicted identification information;

[0156] The value of the third loss function is determined based on the key point location labels and the predicted key point locations;

[0157] The value of the fourth loss function is determined based on the border position corresponding to each category label and the predicted border position corresponding to each predicted category.

[0158] The total loss function value is determined based on the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value.

[0159] The model parameters of the multi-task neural network model are adjusted based on the total loss function value, and training continues until the preset training completion conditions are met, resulting in a well-trained multi-task neural network model.

[0160] This application also provides a computing device to perform the wearable detection method described above. Please refer to... Figure 3 It illustrates a schematic diagram of a computing device provided by some embodiments of this application. For example... Figure 5 As shown, the computing device 7 includes a processor 700, a memory 701, a bus 702, and a communication interface 703. The processor 700, the communication interface 703, and the memory 701 are connected via the bus 702. The memory 701 stores a computer program that can run on the processor 700. When the processor 700 runs the computer program, it executes the wearable detection method provided in any of the foregoing embodiments of this application.

[0161] The memory 701 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this device network element and at least one other network element is achieved through at least one communication interface 703 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0162] Bus 702 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. The memory 701 is used to store programs. After receiving execution instructions, the processor 700 executes the program. The wearable detection method disclosed in any of the aforementioned embodiments of this application can be applied to the processor 700, or implemented by the processor 700.

[0163] The processor 700 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 700 or by instructions in software form. The processor 700 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 701. Processor 700 reads the information in memory 701 and, in conjunction with its hardware, completes the steps of the above method.

[0164] The computing device provided in this application embodiment and the wearable detection method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0165] This application also provides a computer-readable storage medium corresponding to the wearable detection method provided in the foregoing embodiments. Please refer to... Figure 6 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the wearable detection method provided in any of the aforementioned embodiments.

[0166] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0167] The computer-readable storage medium provided in the above embodiments of this application and the wearable detection method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0168] It should be noted that:

[0169] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0170] Similarly, it should be understood that, for the sake of brevity and to aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting a schematic diagram in which the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0171] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0172] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A wearable detection method, characterized in that, include: A multi-task neural network model is used to extract the feature set of each person in the image to be detected. The feature set includes the person's border position, person identification information, head border position, safety helmet border position, work clothes border position, non-work clothes border position, and the upper half of the person's border position. Based on the personnel identification information and the personnel border position, determine the target identifier corresponding to each personnel; Based on the first relative positional relationship between the head frame position and the helmet frame position, it is determined whether each person is wearing a helmet; Based on the second relative positional relationship between the position of the workwear frame and the position of the upper half of the person's body frame, it is determined whether each person is wearing workwear.

2. The method according to claim 1, characterized in that, The multi-task neural network model includes a pyramid network and a head network. The step of using the multi-task neural network model to determine the feature set of each person in the image to be detected includes: The image to be detected is input into the pyramid network, and the target feature map is output after fusing the feature images of different scales corresponding to the image to be detected. The target feature map is input into the head network to obtain the feature set of each person in the image to be detected.

3. The method according to claim 2, characterized in that, The step of inputting the target feature map into the head network to obtain the feature set of each person in the image to be detected includes: The target classification network of the head network determines the category corresponding to each person in the target feature map, the confidence of each category, and the center point position of each person. The category corresponding to each person includes any one of the following: person, head, safety helmet, work clothes and non-work clothes. Based on the confidence level of each category, the target category corresponding to each person is determined from the categories corresponding to each person; The target bounding network of the head network determines the size corresponding to the target category of each person. The head network determines the location of at least one key point for each person through the target key point network, wherein the at least one key point includes at least one of the following: head, left shoulder, right shoulder, left hip, and right hip. Based on the position of at least one key point of each person, determine the position of the upper half of the body frame of each person. The target identifier extraction network of the head network is used to determine the personnel identifier information corresponding to each person; Based on each branch network in the head network, the feature set of each person in the image to be detected is determined.

4. The method according to claim 1, characterized in that, The step of determining the target identifier corresponding to each person based on the person identification information and the person border position includes: Obtain the person identification information and person border position for each person in the detected image at the previous moment; Calculate the similarity between the bounding box position of each person in the detected image at the previous time step and the bounding box position of each person in the image to be detected; The target bounding box position is determined based on the similarity. Output the target bounding box and the target identifier corresponding to the target bounding box.

5. The method according to claim 1, characterized in that, The determination of whether each person is wearing a safety helmet based on the first relative positional relationship between the head frame position and the helmet frame position includes: Calculate the first intersection-union ratio (IUU) of the head frame position and the safety helmet frame position for each person in turn; If the first crossover-union ratio is greater than the first preset crossover-union ratio, then it is determined that the corresponding person is wearing a safety helmet.

6. The method according to claim 1, characterized in that, The determination of whether each person is wearing work clothes based on the second relative positional relationship between the position of the workwear frame and the position of the upper half of the person's frame includes: If the clothing category of the first person is determined to be work clothes, then the second intersection-union ratio of the work clothes border position and the upper half of the person's side border position is calculated. The first person is any one of the multiple people in the image to be detected. If the second crossover ratio is greater than the second preset crossover ratio, then it is determined that the corresponding personnel are wearing work clothes; If it is determined that the clothing category of the first person is work clothes and non-work clothes, then calculate the third intersection-union ratio of the work clothes border position and the upper half of the person's body frame position; If the second crossover ratio is greater than the third preset crossover ratio, then it is determined that the corresponding personnel are wearing work clothes.

7. The method according to claim 1, characterized in that, The multi-task neural network model is obtained through pre-training, and the training process of the multi-task neural network model includes: Obtain a training dataset, which includes multiple sample images, category labels for each person in each sample image, identification information, key point location labels, and bounding box positions for each category label; The training set is input into a multi-task neural network model to obtain the predicted category, predicted identification information, predicted key point position, and predicted bounding box position for each person in each sample image. The value of the first loss function is determined based on the category label and the predicted category; The second loss function value is determined based on the identification information and the predicted identification information; The value of the third loss function is determined based on the key point location labels and the predicted key point locations; The value of the fourth loss function is determined based on the border position corresponding to each category label and the predicted border position corresponding to each predicted category. The total loss function value is determined based on the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value. The model parameters of the multi-task neural network model are adjusted based on the total loss function value, and training continues until the preset training completion conditions are met, resulting in a well-trained multi-task neural network model.

8. A wearable detection device, characterized in that, The device includes: The extraction module is used to extract the feature set of each person in the image to be detected using a multi-task neural network model. The feature set includes the person's border position, person identification information, head border position, safety helmet border position, work clothes border position, non-work clothes border position, and the upper half of the person's border position. The determination module is used to determine the target identifier corresponding to each person based on the person identification information and the person border position; The determining module is used to determine whether each person is wearing a safety helmet based on a first relative positional relationship between the head frame position and the safety helmet frame position; The determining module is used to determine whether each person is wearing work clothes based on the second relative positional relationship between the position of the work clothes frame and the position of the upper half of the person's frame.

9. A computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1-7.

Images